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FOREWORD 


This  report  represents  the  integration  of  three  separate  studies  designed 
to  evaluate  an  instructional  support  system  (ISS)  for  training  F-14  aircraft 
crews.  This  synoptic  approach  had  different  objectives.  The  Allen 
Corporation  was  tasked  to  conduct  an  experimental  study  to  assess  the  training 
effectiveness  and  efficiency  of  the  system,  the  evaluation  was  done  on  site 
at  Naval  Air  Station,  Miramar  under  actual  operating  conditions.  It  involved 
assessment  of  Device  2F95,  an  F-14  Aircraft  Operational  Flight  Trainer,  with 
and  without  the  assistance  of  the  ISS.  (Section  IV,  Instructional  Designer's 
View) 


The  Canyon  Research  Group  obtained  informal  interview  information 
concerning  the  use  of  the  various  instructional  features  of  the  I$$  From  the 
instructor  pilots  of  VF-124.  (Section  III,  Instructor's  View) 


The  Vreuls  Research  Corporation  analyzed  the  data,  evaluated  the 
performance  measurement  capabilities  of  tne  system,  and  integrate  the 
separate  reports  into  this  one.  (fiction  V,  Performance  Measurement  View) 


>y  JOSEPH  A.  PUIG  (j 
Scientific  Officer 
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SECTION  I 


INTRODUCTION 


Current  economic  considerations  create  pressure  to  minimize  use  of 
aircraft  for  training  and  to  increase  the  utilization  of  existing  simula¬ 
tors*  There  is  also  pressure  to  improve  the  quality  of  training  which,  in 
part,  requires  increased  instructional  capability  in  existing  training 
simulators.  These  dual  forces  spawned  the  concept  which  will  be  examined 
in  this  report:  an  Instructional  Support  System  (ISS)  which  can  be  made 
part  of  flight  simulators  without  hardware  or  software  modification  of  the 
original  systems.  The  ISS  provides  a  new  student  and  instructor  interface 
with  the  training  system  which  may  be  used  Instead  of  the  existing 
Instructor  Operator  Station  (IOS)  displays  and  controls;  consequently,  the 
ISS  also  provides  a  vehicle  for  improved  instructional  use  of  training 
simulators. 

The  ISS  development  has  three  specific  goals:  One  is  to  provide  In¬ 
structor  Pilots  with  automation  to  relieve  them  of  ancillary  instructional 
tasks  such  as  system  initialization,  problem  setup,  note  taking,  acting  as 
a  missing  crewmember,  and  serving  as  a  mission  communicator.  Another  is 
to  provide  automated  auxiliary  support  by  incorporating  support  capabili¬ 
ties  typically  not  found  in  current  training  simulators,  such  as  computer- 
resident  syllabi,  computer-generated  briefing,  automated  checkride  testing, 
automated  performance  measurement  and  scoring,  data  files  for  establishing 
performance  norms,  automated  prescriptions,  student  progress  files,  and 
automated  debriefing.  A  third  goal  is  to  provide  a  research  tool  to 
enable  the  solution  of  unresolved  design  issues. 

An  Instructor  Pilot  using  an  F-14  OFT  performs  the  following  functions, 
and  those  which  appear  to  be  candidates  for  automated  support  are  marked 
with  an  asterisk  (from  Semple,  et  al.,  1977): 

*  Review  and  evaluate  the  trainee's  progress  to  date. 

*  Decide  upon  training  content  of  the  simulator  training  exercise  to 
be  undertaken. 

*  Present  a  pre-session  briefing  covering  the  instructional  objectives 
to  be  met  and  the  mission  plan  to  be  used  as  the  training  medium. 

*  Interrogate  the  trainee  on  flight  control,  system  and  operational 
knowledges  required  to  enable  him  to  meaningfully  benefit  from  the  simula¬ 
tor  exercise. 

*  Provide  instruction  in  areas  of  trainee  pre-exercise  knowledge  weak¬ 
nesses. 

*  Provide  over-the-shoulder  instruction  on  cockpit  controls,  displays 
and  procedures. 
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*  Per  fora  plane  captain's  role  by  giving  hand  and  arm  signals  to 
trainee  during  performance  of  engine  start  and  post-start  system  checks. 

*  Perform  missing  crewmember  role  by  reading  NFO  checklists  and  moni¬ 
toring  trainee  responses. 

*  Perform  missing  crewmember  (NFO)  role  by  participating  in  problem 
diagnosis  following  system  failures. 

*  Provide  training  problem  control  in  keeping  with  content  of  the 
Instructor's  Briefing  Guide. 

*  Adjust  training  problem  content  in  keeping  the  trainee's  observed 
performance. 

*  Communicate  system,  procedural  and  operational  knowledge  to  the 
trainee. 

*  Provide  coaching,  cuing  and  performance  feedback  to  the  trainee. 

*  Insert  and/or  remove,  or  command  the  Training  Device  Operator  to 
insert  and/or  remove  system  failures. 

*  Vector  the  trainee  within  a  simulated  warning  area. 

a  Interrogate  the  trainee  on  system  knowledge,  procedures,  causes  and 
consequences  of  failures,  flight  operations,  communication  procedures, 
aircraft  operations  and  operating  limits,  and  NFO  tasks  during  the 
training  exercise. 

*  Take  notes  for  performance  evaluation,  grading/scoring,  learning 
problem  diagnosis,  and  post-exercise  debriefing  of  the  trainee. 

*  Sample  and  evaluate  the  trainee's  performance  in  the  areas  of  system 
knowledge,  normal  procedures,  emergency  procedures,  flight  control,  navi¬ 
gation,  flight  operations  and  voice  communication  procedures. 

*  Complete  and  annotate  grade  sheets. 

*  Perform  communication  functions  such  as:  departure  and  approach 
control,  tower,  GCA  controller,  missed  approach  controller.  Carrier  Air 
Traffic  Control  Center  controller,  Marshall  controller,  Bolter  controller. 
Landing  Signal  Officer  controller,  performance  feedback,  and  mission 
Instructions. 

*  Debrief  the  trainee,  summarizing  strengths  and  weaknesses  as  ascrib¬ 
ing  possible  reasons. 

*  Prescribe  remedial,  extra  and  next  training  content. 

*  Perform  post-exercise  instructional  management  record  keeping. 
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Furthermore,  again  in  the  context  of  the  F-14  OFT,  the  following  Train¬ 
ing  Device  Operator  functions  appear  to  be  candidates  for  partial  or  total 
automated  support: 

*  Complete  system  initialization  by  modifying  or  entering  data  to 
accomplish  emergency  manual  insertion/ removal,  reset,  carrier  site  data 
entry,  ground  site  data  entry,  and  environmental  data  entry. 

*  Activate  and  initialize  the  visual  system. 

*  Respond  to  Instructor  and/or  Trainee  commands  to  accomplish  ground 
power  on/off,  ground  compressed  air  on/off,  remove/insert  (simulated)  wheel 
chocks,  remove/insert  emergencies/failures,  and  operate  slewing  control  to 
reposition  the  simulator. 


The  rationale  behind  the  ISS  concept  is  that  through  automation  of 
functions,  such  as  those  described  above,  both  training  quality  and 
greater  utilization  of  existing  facilities  can  be  realized.  It  is  assumed, 
for  example,  that  reduction  of  instructor  workload  will  lead  to  increased 
training  quality;  automation  of  training  functions  will  lead  to  Improve¬ 
ment  through  standardization  and  centralized  control;  and.  Improvement  of 
simulator  capability  without  an  instructor  present  will  lead  to  greater 
simulator  utilization.  It  is  estimated  that  more  than  20Z  of  Instructor 
Pilot's  (IP)  time  in  simulator  training  is  devoted  to  ancillary  tasks,  and  fre¬ 
quently  the  IP  is  occupied  with  simulator  manipulation  and  cannot  observe 
critical  periods  of  student  performance.  Furthermore,  in  an  environment 
such  as  F-14  Replacement  Pilot  training  (where  the  trainees  already  know 
how  to  fly  some  aircraft),  over  half  of  simulator  utilization  is  logged  as 
"extra  training"  without  an  instructor  present.  The  ISS  concept  is  intended 
to  provide  solutions  in  these  areas. 

An  ISS  was  designed,  developed  and  placed  in  an  operational  environ¬ 
ment  to  test  the  above  concept.  The  ISS  design  and  development  was  guided 
by  a  charter  Intended  to  ensure  that  the  ISS  would  be  responsive  to  both 
operational  training  needs  and  research  needs.  The  charter  is  summarized 
below. 

•  Instructional  support  needs,  as  defined  from  an  operational  training 
reference  point,  would  provide  functional  design  guidance. 

•  The  design  was  to  Incorporate  capabilities  that  will  enable  it  to  be 
used  in  a  research  role  as  well  as  in  an  instructional  role. 

•  A  full  support  system  was  to  be  conceived  with  a  reasonable  subset 
of  capabilities  selected  for  trial  Implementation. 

•  Hardware  and  software  designs  were  to  be  based  on  state-of-the-art 
technology  or  an  extrapolation,  if  associated  risks  were  modest. 
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•  The  design  was  Co  complement  rather  than  unnecessarily  duplicate 
Instructional  features  of  the  host  simulator's  IOS. 

•  System  capabilities  were  to  maximize  the  likelihood  of  user  accep¬ 
tance  and  minimize  user  training  requirements. 

•  Instructional  support  capabilities  were  to  be  modifiable  with  rela¬ 
tive  ease  to  keep  pace  with  the  dynamically  changing  flight  simulator 
training  environment. 

•  The  design  was  to  be  generalizable  to  other  simulator  training 
applications. 

•  The  resulting  system  design  was  to  be  transparent  to  the  host  simu¬ 
lator  and  thus  not  require  modification  to  the  simulator  software. 

The  remainder  of  this  report  is  devoted  to  a  description  of  the  result¬ 
ing  ISS,  tests  which  were  conducted  at  VF-124,  NAS  Miramar,  and  an  analysis 
of  the  ISS  concept  which  emerged.  The  tests  which  are  reported  here  are 
the  work  of  three  different  organizations:  Allen  Corporation,  who  performed 
an  evaluation  from  the  view  of  the  instructional  designer;  Canyon  Research 
Group,  Incorporated,  who  evaluated  from  the  viewpoint  of  the  instructor 
user;  and  Vreuls  Research  Corporation,  who  evaluated  from  the  viewpoint  of 
automated  performance  measurement  design.  A  fourth  organization.  Log! con, 
Incorporated,  designed,  developed  and  Implemented  the  ISS  hardware  and 
software;  their  view,  that  of  the  implementer,  is  presented  as  a  separate 
report  (Kryway  and  Seidenstlcker,  1982). 
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SECTION  II 

IMPLEMENTATION  FOR  DEVICE  2F95 


The  ISS  concept  was  implemented  for  Training  Device  2F95  to  provide  a 
basis  for  test  and  demonstration.  In  addition  to  being  a  functional  train¬ 
ing  aid,  this  ISS  implementation  was  to  be  used  as  a  performance  measure¬ 
ment  and  instructional  research  tool  for  a  broad  range  of  instructional 
support  training  questions.  This  section  provides  a  description  of  the 
implementation;  the  reader  is  referred  to  Seidensticker  and  Kryway  (1982) 
for  a  more  detailed  description. 

DEVICE  2F95 

Device  2F95  is  an  Operational  Flight  Trainer  (OFT)  for  the  F-14A  air¬ 
craft.  It  consists  of  a  simulated  F-14A  pilot  cockpit  which  is  mounted  on 
a  hydraulically  actuated  motion  platform  that  is  capable  of  providing  pitch, 
roll,  heave  and  lateral  displacement  about  the  related  axes.  Visual  scene 
simulation  is  provided  by  a  single  channel,  narrow-f ield-of-view  VITAL 
Ili-S  display  (Device  2B34)  that  provides  the  pilot  with  simulated  out-of- 
the-cockpit  night/dusk  visual  scenes  for  training  land-  and  carrier-based 
takeoffs  and  landings. 

The  F-14  OFT  (i.e.  ,  Device  2F95)  is  designed  for  pilot  training  in  an 
independent  mode,  or  it  can  be  coupled  with  the  Mission  Trainer  (Device 
15C9A)  and  operated  in  a  mission  training  mode.  The  F-14  OFT  incorporates 
a  Xerox  Sigma  5  computer  system  and  is  operated  principally  from  a  remote 
Instructor/operator  station  located  away  from,  but  in  sight  of,  the  simu¬ 
lated  cockpit. 

INSTRUCTOR/OPERATOR  STATION  FOR  DEVICE  2F95 

The  instructor/ operator  station  for  Device  2F95  consists  of  an  instruc¬ 
tor/operator  station,  a  cross-country  and  approach  display  (CCAD)  and  a 
VITAL  IIIA  visual  system  monitor. 

INSTRDCTOR/OPERATOR  STATION.  The  IOS  has  seven  panels.  Each  panel  has 
controls,  indicators,  or  both,  related  to  a  control  or  indicator  in  the 
cockpit.  The  panels  are  grouped  by  function  and  arranged  somewhat  simi¬ 
larly  to  their  related  cockpit  components.  The  controls  allow  the  in¬ 
structor  to  select  or  alter  mission  parameters,  slew  the  aircraft  to  new 
coordinates  or  initial  coordinate  settings,  insert  malfunctions,  and  con¬ 
trol  emergency  conditions. 

The  actual  operation  of  the  OFT  is  accomplished  by  programming  the 
Alphanumeric  Data  Display  (ADD)  system  and  using  the  control  panel  located 
at  the  IOS.  The  ADD  system  has  a  total  of  22  pages  of  programming,  9  of 
which  deal  with  the  parameters  of  the  flight  (i.e.,  carrier  site,  sea 
state,  wind  state,  etc.).  It  is  programmed  using  an  input  terminal  which 
resembles  a  ten  digit  calculator.  By  changing  the  numbers  on  any  of  the 
parameter  pages,  the  instructor  can  design  an  infinite  number  of  different 
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training  experiences  which  includes  an  option  of  184  possible  malfunc¬ 
tions.  Malfunctions  can  be  inserted  Independently  or  in  combination  and 
at  any  time  throughout  the  session.  The  control  panel  contains  controls 
and  displays  that  allow  the  instructor  to  manage  a  training  session.  From 
this  panel,  the  instructor  can  simulate  GCA  conditions,  insert  malfunc¬ 
tions  into  the  session,  control  the  aural  cue  loudness  and  rough  air 
effect,  and  control  the  motion  platform. 

To  effect  a  simulator  lesson,  a  Training  Device  Operator  (TD)  programs 
into  the  ADD  pages  the  lesson  parameters  which  Include  the  initial  setting 
(l.e.  ,  on  the  runway,  6  miles  behind  the  carrier,  etc.),  the  malfunctions, 
wind  conditions,  etc.  From  the  Instructor's  console,  the  Instructor  Pilot 
manages  the  training  session  and  monitors  what  the  pilot  is  doing  in  the 
cockpit.  Color-coded  lights  illuminate  to  Indicate  switch  positions  or 
other  aircraft  parameters  and  an  array  of  repeater  Instruments  provide  the 
Instructor  Pilot  with  information  about  relevant  aircraft  flight  parameters 
and  hence  Information  about  the  performance  of  the  student  flying  the  sim¬ 
ulator. 

CROSS-COUNTRY  AND  APPROACH  DISPLAY  (CCAD).  The  CCAD  is  an  output  device 
that  allows  the  instructor  to  monitor  the  flight  pattern  of  the  trainee 
during  cross-country  or  approach  phases  of  training.  It  consists  of  a  30  x 
30  inch,  vertically-mounted,  rear-projection  display  screen,  a  random  access 
slide  projector,  and  related  controls  and  indicators.  On  the  front  face  of 
the  display  screen  is  an  x-y  plotter  pen  that  plots  the  simulated  flight 
on  the  screen. 

Charts  projected  on  the  screen  are  based  on  aircraft  altitude  and  posi¬ 
tion.  The  indicators  Include  simulated  position,  in  nautical  miles  east  or 
west  and  north  or  south  of  either  a  selected  navigation  facility  or  the 
center  of  the  chart,  and  the  scale  of  the  projected  chart  in  nautical  miles 
per  inch. 

VITAL  DISPLAY.  The  VITAL  IIIA  display  located  at  the  IOS  is  the  same  as 
the  display  mounted  on  the  OFT,  but  without  a  Fresnel  lense.  It  provides 
the  instructor  with  the  same  out-of-the-cockpit  visual  scene  content  as 
that  displayed  to  the  student.  It  provides  a  f ield-of-view  of  approximately 
60  degrees.  Controls  located  on  the  VITAL  display  allow  the  instructor  to 
vary  visibility  and  ceiling  parameters  of  the  simulated  visual  scene. 

INSTRUCTIONAL  SUPPORT  SYSTEM  (ISS) 

The  Instructional  Support  System  (ISS)  is  a  separate  set  of  equipment 
and  computers  attached  to  Device  2F95.  It  controls  operation  of  Device 
2F95  by  generating  inputs  to  the  host  computer  instead  of  the  present  IOS 
console.  The  ISS  hardware  and  software  used  a  transparent  interface  into 
the  present  trainer,  requiring  no  changes  in  present  trainer  hardware  or 
software.  This  allowed  groups  supporting  the  Sigma  5  to  change  their 
software  without  concern  for  interaction  with  the  ISS,  and,  it  allowed 
reversion  to  non-ISS  operation  at  the  flip  of  a  switch. 


10 


NAVTRAEQUIPCEN  81-C-0081-1 


ISS  HARDWARE.  A  block  diagram  of  the  ISS  is  presented  In  Figure  1.  The 
ISS  hardware  consists  of  the  following  equipment:  The  Instructor's 
Consoles  (Primary  and  Secondary),  Printer  Plotter,  Voice  Generation  Unit, 
Applications  Processor  and  Display  Processor. 

Primary  Instructor  Console.  The  primary  instructor's  console  consists  of 
an  upper  and  lower  CRT  display  as  shown  In  Figure  2.  Computer  input  is 
accomplished  on  the  lower  CRT  via  touch  panel  controls.  Training  Is  con¬ 
trolled  from  the  console  by  selecting  options  that  are  displayed  on  the 
lower  CRT  in  "menu-like"  format.  Special  branching  routines  allow  the 
operator  to  branch  to  any  desired  part  of  the  ISS  program.  The  upper  CRT 
is  used  primarily  to  (a)  display  briefing/debriefing  information,  (b)  to 
show  special  GCA/CCA  displays  depicting  both  horizontal  and  vertical  air¬ 
craft  position,  with  respect  to  the  ideal  glldepath,  (c)  aircraft  position 
relative  to  geographical  positions  in  the  form  of  operating  areas,  airways, 
standard  instrument  approaches  and  departures,  and  (d)  text  pertaining  to 
subject  matter  in  the  form  of  aircraft  procedures,  pilot  actions,  diagnos¬ 
tics  pertaining  to  procedures,  and  aircraft  systems  descriptions  when 
called  for  by  the  operator. 

Secondary  Console.  The  secondary,  or  remote,  instructor’s  console  is  iden¬ 
tical  to  the  primary  console.  It  was  located  In  a  room  adjacent  to  the 
Instructor *8  area  of  Device  2F95  and  was  intended  to  be  used  for  training 
session  set-up,  briefing,  playback,  debriefing,  and  performance  grading 
functions.  The  planning,  briefing  and  debriefing  features  of  the  secondary 
console  were  operable  simultaneously  with  the  conduct  of  a  trainer  mission; 
however,  use  of  the  secondary  console  slowed  response  at  the  primary  con¬ 
sole. 

Printer  Plotter.  The  printer  plotter  functioned  as  a  hardcopy  device  for 
instructional  purposes  and  as  a  printer  for  maintenance  and  administrative 
purposes. 

Voice  Generation  Unit.  The  voice  generation  unit  provided  a  computer¬ 
generated  simulation  of  the  controlling  agencies  a  pilot  normally  would 
communicate  with  during  actual  flight.  This  unit  provided  voice  transmis¬ 
sions  to  the  cockpit  in  the  form  of  information,  Instructions  and  clear¬ 
ances  With  this  implementation,  all  transmissions  were  made  with  one 
same-sounding  voice. 

Applications  Processor.  The  applications  processor  consisted  of  computer 
equipment  from  Data  General  Corporation  including  (a)  an  Eclipse  central 
processing  unit  with  196k,  16-bit  words  of  main  memory,  (b)  a  96  megabyte 
disk  storage  unit,  (c)  an  industry  standard  half-inch  magnetic  tape  unit, 
and  (d)  various  interfaces  with  peripheral  equipment  and  a  link  to  the 
Display  Processor. 

The  Applications  Processor  controlled  the  operation  of  the  ISS  in  re¬ 
sponse  to  direction  from  operator  touch  panel  Inputs  and  activity  on  Device 
2F95  using  data  and  programs  stored  on  the  disk. 
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INSTRUCTOR ' S  REMOTE 
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Figure  1.  ISS  System  Block  Diagram. 
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Display  Processor.  The  Display  Processor  consisCed  of  computer  equipment 
from  Data  General  Corporation  including  (a)  an  Eclipse  central  processing 
unit  with  128K,  16-bit  words  of  main  memory,  (b)  a  10  megabyte  disk  storage 
unit,  and  (c)  a  link  to  the  Applications  Processor.  The  Display  Processor 
generated  and  controlled  the  Images  which  appeared  on  the  operator  console 
in  response  to  display  requirements  received  from  the  Application  Processor 
and  operator  commands  received  from  the  touch  panels. 

ISS  FUNCTION  AND  OPERATION.  The  ISS  was  designed  to  provide  the  instructor/ 
operator  with  a  choice  of  three  modes  of  operation:  CANNED,  Instructor 
select  (ISEL) ,  and  special  task  training  (STT). 

CANNED  Mode.  The  CANNED  Mode  is  a  completely  automated  mode  of  operation 
whereby  training  objectives  in  the  form  of  task  modules  have  been  prese¬ 
lected  and  programmed  to  conform  to  a  formalized  training  syllabus.  Oper¬ 
ating  in  the  CANNED  mode,  the  ISS  follows  a  planned  flight  course,  inserts 
and  removes  aircraft  malfunctions,  gives  required  voice  communications, 
and  calculates  and  assigns  grades  for  each  task  objective. 

ISEL  Mode.  ISEL  Mode  is  a  mode  of  operation  whereby  the  instructor  has 
the  option,  during  the  Training  Session  Set-up  phase,  of  constructing  a 
training  session  with  the  use  of  task  modules  to  meet  his  own  tailored 
training  objectives. 

STT  Mode.  The  STT  Mode  is  mode  of  operation  designed  to  be  used  for  spe¬ 
cialized  part  task  training  of  specific  training  objectives.  It  allows  a 
training  scenario  to  be  constructed  from  a  list  of  specific  flight  maneu¬ 
vers  such  as  GCA/CCA,  ILS  approaches,  bolter  patterns,  catapult  takeoff, 
etc.  The  STT  Mode  allows  repeated  practice  of  individual  flight  maneuvers 
through  an  automated  re-position  capability  that  frees  the  pilot  from  fly¬ 
ing  the  simulator  to  the  Initialization  point  for  each  maneuver. 

TASK  MODULE  CONCEPT.  An  analysis  of  the  structure  of  FRS  and  probably  UPT 
flight  training  reveals  that  the  students'  tasks  break  into  functional 
groups  with  specific  training  objectives.  A  task  module,  in  the  context 
of  ISS  design,  is  a  logically  related  series  of  external  events  and  pilot 
actions  that  should  result  in  the  achievement  of  predefined  training  cri¬ 
teria.  For  example,  a  task  module  may  be  the  takeoff  from  a  particular 
ground  or  carrier  base,  a  pre-start  checklist,  or  a  wing-sweep  malfunction. 
Normal  task  modules  encompass  those  tasks  related  to  pre-  and  post-flight 
checkouts  and  checklists  during  flight.  Flight  task  modules  encompass 
those  tasks  related  to  flying  skills,  procedures  and  navigation.  Malfunc¬ 
tion  task  modules  relate  to  system  failures. 

Tasks  modules  have  important  characteristics.  They  can  be  executed,  and 
have  well-defined  beginning  and  ending  points.  They  can  be  run  in  parallel, 
independent  of  each  other.  They  form  the  basis  of  high-level  simulator 
control,  exercise  definition  and  replay.  They  provide  the  vehicle  for 
associated  training  functions,  such  as  performance  evaluation  and  record 
keeping.  A  more  complete  discussion  of  the  task  module  concept  may  be 
found  in  Seidensticker  and  Meyn  (1982). 
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PERFORMANCE  MEASUREMENT  AND  (StADING.  The  ISS  perforaance  monitoring  mea¬ 
surement  and  grading  are  discussed  below. 

Event  Monitoring.  A  program  dedicated  to  monitoring  events  is  notified 
when  a  task  module  becomes  active,  and  a  file  is  opened  which  defines  the 
specific  events  appropriate  to  that  task  module.  Those  events  are  then 
monitored  and  tested  for  specific  relationships  and  contingencies,  also 
defined  in  the  task  modules.  Therefore,  when  an  event  occurs,  alternate 
actions  may  take  place  depending  on  the  circumstances  present  at  the  time. 
For  example,  a  diagnostic  message  can  be  generated  if  the  aircraft  drifts 
off  the  center  of  the  runway  during  takeoff. 

Performance  Measurement.  One  of  the  actions  that  may  be  taken  as  a  result 
of  an  event  is  the  calculation  of  measurements  of  flight  parameters.  A 
performance  measurement  program  is  dedicated  to  taking  measurements  for 
each  task  module  that  becomes  active.  These  measurements  include  measure¬ 
ments  calculated  from  the  data  obtained  by  sampling  flight  parameters  1  to 
20  times  per  second  during  a  prespecified  mission  interval,  continuous 
measurement  whenever  specified  limits  are  exceeded,  or  a  single  parameter 
(snapshot)  sampled  when  a  specific  event  occurs. 

Procedure  Evaluation.  Performance  measurement  for  normal  and  emergency 
procedures  is  accomplished  differently  than  that  for  flight  parameters. 
The  event  monitoring  program  is  capable  of  evaluating  a  sequence  of  events 
such  as  activating  switches  in  a  particular  sequence,  and  activating  other 
switches  in  no  particular  sequence,  but  after  the  other  switches  which  had 
to  be  sequenced  in  order.  For  these  kinds  of  situations,  the  following 
measures  can  be  calculated:  critical  errors,  recognition  reaction  time, 
total  procedure  time,  percent  of  mandatory  actions  taken,  and  percent  of 
optional  actions. 

Scoring.  Scores  are  computed  when  a  task  module  is  completed.  A  scoring 
template  is  defined  along  with  each  task  module  that  includes  all  measure¬ 
ment  definitions  and  scoring  formulae.  The  template  Includes  allowable 
scores  (e.g.  4.0,  3.5,  3.0,  2.5)  and  the  range  of  values  for  a  measurement 
for  each  scoring  category.  A  measure  that  falls  within  one  of  these  ranges 
is  assigned  the  corresponding  score.  An  example  of  such  a  template,  as  it 
could  be  displayed  to  the  instructor,  is  shown  in  Figure  3.  Additional 
information  on  procedures  monitoring  and  scoring  in  the  ISS  may  be 
found  in  Seidens ticker  and  Kryway  (1982). 
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GRADING  CRITERIA 

SAN  PEDRO  DEPART  FINAL  GRADE  -  2.7 


MEASURE  NOMINAL  4.0  3.5  3.0  2.5  VALUE  GRADE  WT 


DME  SNAPSHOT 

1 

2.5 

2.75 

3 

.79 

.10 

1 

0 

2.51 

2.76 

3.01 

S* 

RADIAL  DEVIATION (RMS ) 

280. 

2 

3 

5 

10 

.69 

rail 

.10 

280. 

0 

2.01 

3.01 

5.01 

ALTITUDE  DEVIATION(RMS) 

2000. 

100. 

200. 

300. 

500. 

206.96 

3.0 
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■ 
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■ 
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1 

Figure  3.  Example  Scoring  Template. 
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SECTION  III 


INSTRUCTORS’  VIEW 


METHOD 

The  general  methods  used  were  interview  and  observation,  A  sample  of 
VF-124  instructors  who  instructed  using  Device  2F95  were  interviewed  prior 
to  the  activation  of  the  ISS  at  VF-124.  Post-use  interviews  were  conducted 
with  a  sample  of  Instructor  Pilots  after  they  had  gained  some  experience 
with  the  ISS.  The  ISS  also  was  observed  during  operation.  Pre-use  and 
post-use  interviews  are  discussed  more  fully  below. 

PRE-USE  INTERVIEWS.  The  purpose  of  the  pre-use  interviews  was  to  identify 
any  strong  biases,  either  for  or  against  the  concepts  and  technologies 
reflected  in  the  ISS.  The  interview  guide  shown  in  Appendix  A  was  used. 
Each  instructor  who  was  interviewed  was  given  a  short  briefing  on  the  experi¬ 
mental  nature  of  ISS  and  the  fact  that  ISS  was  a  technology  demonstration 
and  evaluation  project.  General  features  and  capabilities  of  ISS  were 
summarized.  Interviews  then  were  completed  using  the  interview  guide. 

Two  interviewers  used  the  guide  to  interview  12  of  approximately  20 
Instructor  Pilots  who  possibly  could  have  used  the  F-14  OFT  for  instruc¬ 
tion.  Only  four  of  the  Instructors  who  were  interviewed  ultimately  used 
the  prototype  experimental  ISS.  It  was  not  known  at  the  time  of  the  pre¬ 
use  interviews  which  instructor  would  eventually  use  the  ISS. 

POST-USE  INTERVIEWS.  The  purpose  of  the  post-use  interviews  was  to  obtain 
user  feedback  on  implementation,  instructional  value,  ease  of  use  and  general 
acceptance  of  ISS  concepts  and  capabilities.  A  single  interviewer  conducted 
the  interviews  using  the  guide  presented  in  Appendix  B.  The  interviewer 
had  participated  in  the  functional  design  of  ISS  (Semple  et  al.,  1979)  and 
was  quite  familiar  with  the  system's  capabilities. 

Post-use  interviews  were  conducted  at  two  different  time  periods:  Sep¬ 
tember  1981  and  January  1982.  Four  Instructor  Pilots  were  Interviewed  in 
September,  and  six  were  interviewed  in  January.  Of  the  six  interviewed  in 
January,  two  previously  had  been  interviewed  in  September.  Two  additional 
instructors  who  had  gained  experience  with  ISS  were  not  available  for  inter¬ 
view  in  either  period. 

RESULTS 

PRE-USE  INTERVIEWS.  A  majority  of  instructors  interviewed  had  prior  exper¬ 
ience  with  computer-based  systems  other  than  aircraft  avionics.  Their  views 
toward  computer-based  instructional  assists  appeared  highly  objective.  The 
consensus  was  that  the  nature  of  the  computer  programs  and  the  man-machine 
interface  would  determine  the  value  of  ISS-like  systems.  A  general  concern 
was  expressed  that  computer-based  instructional  supports  might  unnecessarily 
limit  the  flexibility  that  all  felt  was  needed  for  transition  training 
using  an  operational  flight  trainer.  More  complete  information  on  student 
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and  system  performance  was  seen  as  a  benefit,  as  long  as  the  information 
was  made  available  to  instructors  in  readily  usable  forms  and  formats.  A 
majority  felt  that  systems  like  ISS  would  not  be  difficult  to  use  because 
of  the  state-of-the-art  in  man-computer  interface  design.  Mixed  reactions 
were  expressed  regarding  whether  ISS  would  simplify  the  instructor's  job, 
and  none  expressed  the  view  that  ISS  would  result  in  improvements  to  their 
instructional  skills.  Considerable  doubt  was  expressed  regarding  whether 
computer  models  of  air  traffic  or  final  approach  controllers  would  be 
practical.  Strong  objection  was  raised  to  computer-generated  performance 
scores.  Those  who  did  not  object  cited  potential  improvements  to  objec¬ 
tivity  and  standardization.  Most,  however,  objected  on  the  following 
grounds.  The  simulator  ''flies"  differently  from  the  aircraft,  and  there¬ 
fore,  simulator  performance  scores  would  be  invalid.  The  same  instructors 
questioned  the  "flight”  training  value  of  Device  2F95,  particularly  with 
respect  to  precise  flight  path  control.  Computer-based  scoring  is  too 
rigid.  Much  of  flying  is  "subjective"  and  focuses  on  decision  making, 
which  they  felt  was  not  amenable  to  automated  scoring.  Several  expressing 
these  views  also  indicated  that  they  might  be  less  apprehensive  after 
using  computer-generated  performance  scores.  A  majority  also  expressed  a 
pre-use  view  that  instructor  station  information  displays  should  include 
cockpit  instrument  forms,  at  least  for  flight  control,  propulsion  system 
and  any  trend  information.  A  "wait  and  see"  attitude  characterized  res¬ 
ponses  responses  to  the  acceptability  of  the  touch  panel  information  entry 
characteristic  of  ISS. 

POST-USE  INTERVIEWS.  The  following  text  presents  instructor  opinions  and 
viewpoints  to  ISS  topics  not  directly  addressed  in  Chapter  IV. 

Overall  Impressions.  Not  all  of  the  instructors  who  used  the  experimental 
prototype  ISS  had  the  opportunity  to  gain  desired  levels  of  experience  and 
proficiency  with  the  system.  Additionally,  not  all  received  adequate  In¬ 
struction  on  the  system's  purposes,  capabilities,  options  and  limits.  Their 
viewpoints  and  opinions  must  be  interpreted  in  this  general  context. 

A  majority  of  Instructors  who  used  ISS  viewed  most  of  the  instructional 
support  and  control-display  concepts  embodied  in  the  system  as  sound.  Excep¬ 
tions  are  noted  below.  Also,  a  majority  expressed  the  opinion  that  ISS 
technologies  present  a  solid  basis  around  which  to  build  future  instructor/ 
operator  stations  and  support  systems  for  simulator  training.  Computer 
graphics  displays  were  well  received  because  they  provided  pictorial 
representations  of  flight.  Consolidations  of  control  options  and  display 
content  into  highly  focused,  easily  scanned  areas  also  were  commented  upon 
favorably. 

On  the  negative  side,  perceived  reductions  in  instructor  options  and 
Instructional  flexibility  were  viewed  with  disfavor.  Automated  malfunction 
Insertion  and  removal,  together  with  automated  performance  scoring,  were 
found  to  be  at  the  forefront  of  this  objection.  Additionally,  several 
Instructor  Pilots  expressed  the  view  that  simply  adding  ISS-like  capa¬ 
bilities  to  existing  instructor/operator  stations  likely  would  not  be 
productive.  This  view  reflected  a  lack  of  understanding  on  the  part  of 
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some  that  the  experimental  prototype  ISS  was  Intended  largely  for  tech¬ 
nology  exploration  and  refinement,  rather  than  an  adjunct  to  existing 
simulator  consoles. 

Use  of  Existing  IQS  Displays.  The  experimental  prototype  ISS  station  was 
not  intended  to  fully  replace  the  existing  Instructor/operator  station. 
Rather,  it  was  intended  and  designed  to  replace  some  functions,  add  func¬ 
tional  capabilities,  and  only  complement  others.  For  example,  the  ISS 
design  did  not  attempt  to  duplicate  or  displace  flight  control  or  pro¬ 
pulsion  Information  displays  on  the  existing  console.  Configuration  infor¬ 
mation,  Including  gear,  flap  and  wing  position,  were  displayed  on  ISS.  All 
instructors  interviewed  indicated  that  they  did  refer  to  flight,  con¬ 
figuration  and  propulsion  information  displays  on  the  existing  console. 
Several  commented  that  they  referred  to  the  existing  console  much  less 
frequently  then  they  thought  they  would.  Only  one  interviewee  expressed 
the  opinion  that  the  ISS  provided  no  information  which  was  not  directly 
available  pr  estimable  from  the  existing  instructor/operator  console.  All 
others  felt  that  the  ISS  provided  additional,  useful  Information. 

Display  Viewing  Comfort.  All  instructors  who  were  interviewed  indicated 
that  neither  the  graphics  nor  alphanumeric  displays  produced  any  noticeable 
visual  fatigue.  Additionally,  all  displayed  information  was  easily  readable. 
Glare  also  was  not  a  problem. 

Touch  Panel  Control  Medium.  Instructor  inputs  to  ISS  were  made  using  a 
CRT  touch  panel.  Instructors  found  this  control  medium  quite  acceptable, 
but  did  not  strongly  favor  it  over  rotary  switches,  pushbuttons,  keyboards 
or  other  control  devices.  A  majority  commented  that  controls  were  much 
more  centralized  and  easier  to  reach  using  the  touch  panel. 

The  ISS  touch  panel  mechanization  resulted  in  several  instructor 
complaints.  One  involved  registration.  Several  touches  on  or  near  the 
target  often  were  required  to  achieve  control  activation.  When  a  control 
input  called  for  a  new  information  display,  an  additional  problem  was  that 
several  seconds  could  elapse  between  the  request  for  the  display  and  the 
presentation  of  the  display.  If  the  control  target  was  again  touched  in 
this  lag  period,  a  different  and  undesired  input  to  ISS  could  result. 
Both  of  these  problems  can  be  overcome  through  different  mechanization. 

One  aspect  of  making  the  touch  panel  control  input  approach  acceptable 
appears  to  be  providing  all  immediately  necessary  control  capabilities  on 
each  control  page  that  is  displayed.  This  is  needed  to  avoid  having  to 
page  through  numerous  displays  to  get  the  desired  control  input  capability. 
ISS  appeared  to  have  achieved  this  goal,  which  is  highly  desirable. 

Automated  Checklists.  The  automated  checklist  displays  presented  procedural 
sequences  that  student  pilots  should  perform,  for  example,  during  engine 
start.  Clock  times  at  which  each  procedure  was  performed  were  displayed 
beside  a  text  readout  of  each  procedural  step.  This  allowed  the  instructor 
to  monitor  the  sequence  in  which  procedures  actually  were  accomplished. 
Additionally,  certain  improper  procedures  were  displayed  as  "diagnostic" 
information. 
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This  design  characteristic  was  one  of  the  most  well  accepted  by  the 
instructors  who  were  interviewed.  The  information  that  was  displayed  was 
viewed  as  valuable  as  was  well  in  addition  to  similar  Information  available 
through  the  existing  Instructor/operator  station.  Clock  times,  per  se,  were 
not  believed  to  be  of  value.  Rather,  the  importance  of  this  indication  was 
felt  to  lie  in  providing  the  instructor  with  a  visual  indication  of  proce¬ 
dural  sequences.  The  potential  value  of  using  hardcopy  printouts  of  auto¬ 
mated  checklist  information  for  debriefing  remains  unknown. 

Aircraft  Systems  Information  Callups.  The  ISS  contained  several  aircraft 
system  information  files  that  instructors  could  call  up  for  display  via 
the  runtime  menu.  Information  displayed  to  the  instructors  was  both  graphic 
and  textual,  and  was  taken  directly  from  relevant  pages  of  the  F-14  NATOPS 
manual. 

Three  instructors  used  the  system  information  callups.  All  three  found 
the  information  of  value  for  instruction.  The  level  of  detail  involved 
exceeds  typical  human  long  term  memory  characteristics.  Also,  a  copy  of 
the  needed  NATOPS  manual  pages  frequently  is  not  available  at  the  training 
device. 

Based  upon  the  positive  instructor  response  to  the  memory  aid  value  of 
the  systems  information  callups,  it  appears  desirable  to  consider  similar 
information  callups  in  future  instructor/operator  station  applications. 
If  provisions  for  similar  information  display  are  made  in  future  devices, 
provisions  for  easy  modification  of  information  content  also  must  be  made. 
This  follows  since,  for  example,  the  content  of  NATOPS  manuals  changes 
quite  often,  and  information  stored  in  an  instructional  support  system 
could  become  quickly  outdated. 

Digital  Readouts.  The  ISS  graphics  display  presented  digital  readouts  of 
selected  information  elements  such  as  heading,  airspeed  and  communication/ 
navigation  channels  selected  by  the  simulator  pilot.  Practically  none  of 
the  instructors  interviewed  recalled  having  seen  or  used  the  digital  read¬ 
outs.  Practically  all  indicated  a  preference  for  cockpit-type  instruments 
at  the  instructor/operator  station,  probably  since  they  are  most  familiar 
with  these  display  types.  Additionally,  almost  all  expressed  concern 
about  using  digital  readouts  to  assess  trend  information.  Several  com¬ 
mented,  however,  that  the  concept  appeared  interesting  from  the  standpoint 
of  concentrating  a  number  of  information  elements  into  a  relatively  com¬ 
pact  area.  They  noted  that  this  would  reduce  scanning  requirements. 

From  an  historical  standpoint,  displaying  the  communication  and  naviga¬ 
tion  channels  selected  by  the  simulator  pilot  was  done  to  solve  particular 
problems.  The  existing  OFT  instructor/operator  console  does  not  display 
these  information  elements.  This  has  resulted  in  a  number  of  cases  where 
the  instructor  and  student  were  not  coordinated  in  navigation  and  communi¬ 
cation  tasks.  This  detracted  from  available  simulator  training  time  and, 
possibly,  simulator  training  value. 

Programmed  Malfunctions.  ISS  had  the  capability  in  both  the  CANNED  and  ISEL 
modes  to  automatically  insert  and  remove  malfunctions.  About  midway  through 
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the  evaluation,  the  ISEL  mode  was  modified  to  enable  manual  malfunction 
insertion  and  removal* 

None  of  the  instructors  reacted  favorably  to  automatic  malfunction 
insertion  or  removal*  The  following  reasons  were  given.  Programmed  mal¬ 
functions  were  too  rigid  and  the  selection  and  timing  of  malfunctions  were 
not  under  instructor  control.  If  a  student  pilot  had  not  effectively 
coped  with  an  ongoing  malfunction,  burdening  him  with  a  second  malfunction, 
somewhat  concurrently,  was  felt  to  interfere  with  instructor  intervention. 
Some  malfunctions  were  automatically  removed  before  they  were  handled 
effectively.  The  fact  that  they  were  not  handled  effectively  could  have 
caused  subsequent  system  failures.  Clearly,  this  would  pose  an  undesirable 
training  event  with  potentiallly  undesirable  operational  consequences. 

The  need  to  allow  instructors  to  arbitrarily  change  their  minds  on  which 
malfunctions  should  be  inserted  was  not  well  justified  by  the  instructors, 
aside  from  the  fact  that  the  existing  console  enables  them  to  do  so.  The 
other  points  are  valid.  It  should  be  noted,  however,  that  the  ISS  concept 
allowed  for  the  occurrence  of  subsequent,  related  malfunctions  if  the  one 
inserted  was  not  effectively  handled. 

The  midway  change  to  the  ISEL  mode  allowed  Instructor  selection  of  mal¬ 
functions  and  control  over  their  insertion  and  removal.  The  modification 
was  well  received  the  instructors  who  used  it.  Similar  approaches  on  other 
training  simulators  have  been  well  received  by  instructors  (Semple,  et 
al. ,  1981).  In  these  systems,  malfunctions  to  be  used  during  a  training 
session  are  selected  from  an  electronically  generated  shopping  list. 
Those  selected  then  can  be  displayed  during  training,  and  easily  inserted 
and  removed  at  the  instructor's  discretion. 

Automated  Performance  Scoring.  ISS  incorporated  automated  performance 
measurement  capabilities  for  developing  task  module  performance  scores. 
Scoring  was  possible  for  both  procedural  and  flight  modules.  Scores  were 
generated  using  best  estimate  scoring  algorithms  which  had  not  been  sub¬ 
jected  to  empirical  research  and  development.  The  scores  that  were  pro¬ 
duced  were  normalized  to  a  scale  of  one  through  four,  corresponding  with 
the  grading  scale  used  by  VP-124.  In  the  initial  ISS  configuration,  task 
module  performance  scores  were  displayed  to  the  instructor  only  at  the 
conclusion  of  training,  for  use  during  debriefing.  At  that  time  instruc¬ 
tors  also  could  call  up  the  performance  measurement  details  chat  went  into 
the  makeup  of  any  score.  Approximately  midway  through  the  evaluation,  ISS 
was  modified  to  also  display  performance  scores  at  the  conclusion  of  each 
task  module  (e.g.  during  training).  During  the  debriefing  mode,  instruc¬ 
tors  could  change  any  performance  score  through  the  ISS  console. 

Several  of  the  instructors  Interviewed  either  did  not  observe  the  ISS- 
generated  performance  scores  or  felt  they  had  too  little  experience  with 
them  to  comment  validly.  Those  who  had  some  experience  with  the  scores 
commented  that  the  scores  did  not  appear  valid.  This  was  to  be  expected 
since  the  scoring  algorithms  being  used  had  not  been  tested  or  refined. 
A  majority  of  those  who  viewed  the  s«  ores  expressed  doubts  that  computer 
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scoring  was  applicable  to  piloting  skills.  Others  felt  the  concept  was  too 
rigid  and  would  preclude  Instructor  Inputs.  Anticipation  of  the  latter 
concern  was  one  reason  for  allowing  Instructors  to  change  computer  genera¬ 
ted  scores  in  the  ISS  debriefing  mode.  None,  however,  used  ISS  in  the 
debriefing  mode  and  so  they  were  unaware  of  this  capability.  A  minority 
of  instructors  interviewed  indicated  a  willingness  to  try  out  more  fully 
developed  performance  scores  before  rendering  a  final  opinion. 

Hardcopy  Printouts.  ISS  was  capable  of  providing  hardcopy  printouts  of 
various  alphanumeric  and  graphic  Information  displayed  at  the  ISS  instruc¬ 
tor's  console.  The  hardcopy  capability  was  not  used  by  any  of  the  in¬ 
structors  interviewed.  One,  however,  had  used  hardcopy  printouts  of  GCA 
approach  profile  graphics  from  a  training  simulator  he  had  used  on  a  prior 
assignment.  He  indicated  that  the  graphics  printouts  were  useful  during 
debriefing.  Similar  uses  of  graphics  hardcopy  printouts  have  been  reported 
by  Semple,  Cotton  and  Sullivan  (1981).  Neither  the  present  study  nor 
prior  studies,  however,  have  been  able  to  address  the  value  of  other  types 
of  hardcopy  printouts,  such  as  performance  scores  or  checklist  procedural 
details,  for  debriefing.  The  value  of  hardcopy  printouts  as  debriefing 
aids  requires  additional  Investigation. 
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SECTION  IV 

INSTRUCTIONAL  DESIGNERS  VIEW 


PURPOSE 

The  purpose  of  Che  Investigation  described  in  this  section  was  to  assess  the 
contribution  of  the  ISS  to  the  training  and  operational  efficiency  of 
Device  2F95.  This  assessment  took  the  form  of  an  experimental  comparison 
of  the  effectiveness  and  efficiency  of  the  training  provided  by  Device 
2F95  with  and  without  the  assistance  of  the  ISS.  More  specifically,  data 
were  collected  for  the  following  measurement  categories:  training  effec¬ 
tiveness,  training  efficiency,  standardization  of  training,  effectiveness 
and  utilization  of  special  instructional  feature,  and  general  measure  of 
ISS  and  Device  2F95  IOS  operation  and  performance. 

METHOD 

SUBJECTS.  The  personnel  who  served  as  subjects  for  the  study  were  student 
pilots  undergoing  training  at  Fleet  Replacement  Squadron  (FRS)  VF-124  loca¬ 
ted  at  NAS  Miramar,  California.  Student  pilots  were  categorized  according 
to  their  previous  flying  experience.  Category  I  refers  to  pilots  who  have 
only  completed  Undergraduate  Jet  Pilot  Training  (UJPT),  or  pilots  who,  after 
completing  UJPT,  served  as  UJPT  Instructor  pilots  before  being  assigned  to 
the  FRS.  Category  II  refers  to  pilots  who  normally  have  between  6 00-1000 
hours  of  total  flight  time  and  that  have  fleet  fighter  experience  in  an 
aircraft  other  than  the  F-14.  Category  III  refers  to  pilots  who  have 
operational  experience  in  the  F-14. 

Overall,  thirty-three  student  pilots  participated  in  the  study.  Twenty 
students  from  classes  8101,  8102  and  8103  took  part  in  the  CQ  portion  of 
training.  Twelve  of  these  students  were  Category  I  pilots,  six  were  Category 
II  pilots  and  two  were  Category  III  pilots.  Two  of  the  Category  I  pilots 
repeated  CQ  training  and  were  included  in  the  study  both  times  because  it 
was  felt  that  the  previous  CQ  training  would  not  affect  the  measures  being 
collected.  Thus,  data  were  collected  for  the  equivalent  of  22  students 
undergoing  CQ  training. 

Thirteen  students  from  classes  8107,  8108  and  8201  participated  in  the 
FAM  training  portion  of  the  study.  Twelve  of  these  students  were  Category 
I  pilots  and  one  was  a  Category  III  pilot. 

INSTRUCTORS.  Seventeen  Instructor  Pilots  with  different  levels  of  experi¬ 
ence  participated  in  the  study  and  were  those  normally  assigned  by  VF-124. 
A  brief  orientation  course  on  how  to  operate  the  ISS  and  on  the  purpose  of 
the  study,  was  provided  to  some  VF-124  personnel  before  the  study  began. 
Several  training  sessions  on  how  to  operate  the  ISS  also  were  scheduled 
prior  to  the  study.  However,  many  of  the  instructors  who  participated  in 
the  study  were  neither  available  for  the  orientation  nor  the  training. 

Instructor  training  on  how  to  operate  the  ISS  typically  was  given 
"on-the-job"  by  Logicon  personnel.  It  normally  began  the  first  time  an 
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instructor  arrived  to  conduct  a  training  event  that  was  scheduled  to  be 
given  with  the  assistance  of  the  ISS.  Students  who  participated  in  CQ 
training,  and  underwent  ISS  training,  were  trained  to  use  the  ISS  in  a 
similar  manner.  (All  of  the  ISS  controlled  CQ  trainer  events  were  self- 
instructed). 

STUDY  DESIGN.  The  study  was  designed  to  assess  the  contribution  of  the 
ISS  to  the  effectiveness  and  efficiency  of  Device  2F95  training.  This 
involved  having  two  separate  groups  of  students  undergo  training;  a  control 
group  which  received  their  training  with  the  standard  IOS  in  operation  and 
an  experimental  group  which  received  their  training  with  the  assistance  of 
the  ISS.  Experimental  and  control  groups  were  formed  separately  for  both 
FAM  training  and  CQ  training  for  a  total  of  four  groups.  Students  were 
assigned  to  experimental  and  control  groups  in  a  manner  that  matched,  as 
nearly  as  possible,  the  amount  and  type  of  previous  flying  experience. 

Category  I  pilots  who  had  previous  UJPT  instructor  experience  were 
matched  on  the  basis  of  type  of  IP  experience  (T-2C  or  TA-4J),  whereas 
students  were  matched  on  the  basis  of  overall  UJPT  grades.  Category  II 
pilots  were  matched  on  the  basis  of  previous  type  of  fighter  experience. 
Students  were  followed  through  their  training  one  class  at  a  time  and 
therefore  were  matched  within  their  own  class.  Overall,  eleven  students 
each  were  assigned  to  the  control  and  experimental  groups  for  CQ  training. 
For  familiarization  (FAM)  training,  seven  students  were  assigned  to  the 
control  group  and  six  students  to  the  experimental  group. 

Date  were  collected  on  experimental-  and  control-group  students  under¬ 
going  ti  'ining  on  Device  2F95  for  the  Familiarization  events:  FPT  010,  FAPT 
020,  FA  T  030  and  FAPT  040.  Likewise,  data  were  collected  for  Carrier 
Quallfii  ation  events:  CQPT  010,  CQPT  020  and  CQPT  030.  In  addition,  data 
were  collected  for  simulator  qualification  events  FAPT  050  and  CPT  040. 
However,  these  two  events  were  always  trained  with  the  IOS  regardless  of 
the  previous  group  assignment. 

DATA  COLLECTION  PROCEDURES.  Students  were  scheduled  for  their  simulator 
training  according  to  the  syllabus  requirements  of  VF-124.  All  students 
received  the  same  training  for  each  particular  stage  of  training.  The  only 
difference  between  the  training  received  by  control  and  experimental  stu¬ 
dents  was  the  console  (IOS  or  ISS)  from  which  training  was  controlled. 

Students  were  monitored  and  data  were  collected  during  briefing  and 
debriefing  periods  as  well  as  during  actual  trainer  activities.  When  the 
student/instructor  arrived  at  the  trainer  area  for  their  first  FAM  or  CQ 
training  event,  they  were  provided  ISS  training,  as  required,  and  informed 
of  the  purpose  of  the  study.  It  was  emphasized  that  the  intent  of  the 
study  was  not  to  evaluate  either  student  or  instructor  performance  per  se, 
but,  instead,  to  evaluate  the  operational  aspects  of  the  instructor's  sta¬ 
tion. 

Instructors  and  students  were  informed  that  data  collection  and  obser¬ 
vation  would  be  discontinued  at  their  request  if  they  thought  it  would 
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interfere  with  training.  This  option  was  exercised  only  once  when  the 
data  collector  was  asked  not  to  attend  a  debriefing,  or  copy  the  grades, 
of  a  student  who  received  a  "down"  on  a  CQ  trainer  event  (CQPT  00). 

Multiple  measures  were  collected  on  the  way  in  which  training  was  con¬ 
ducted  for  I0S-  and  ISS-controlled  training.  They  included: 

•  Amount  of  time  spent  briefing  the  event  to  be  trained. 

•  Amount  of  time  taken  to  set-up  or  initialize  the  trainer. 

•  Amount  of  time  simulator  was  actually  used  for  training. 

a  Number  and  type  of  syllabus  events  completed  per  training  event. 
(CQ  only) 

o  Number  and  type  of  special  Instructional  features  used  for  briefing, 
actual  training,  and  debriefing. 

•  Number  of  times  IOS/ISS  had  to  be  re-initialized  due  to  10S/ISS  mal¬ 
function. 

o  Number  of  times  IOS/ISS  training  terminated  due  to  IOS/ISS  malfunc¬ 
tion. 

•  Number  and  type  of  aircraft  malfunctions  inserted. 

o  Number  and  type  of  errors  made  by  IPs  and  TDs  in  operating  instruc¬ 
tor’s  console. 

o  The  amount  of  time  spent  debriefing. 

o  ISS-generated  grades  of  student  performance. 

o  IP  grades  of  student  performance. 

These  measures,  except  for  grades,  were  collected  on  data  collection 
forms  developed  especially  for  the  evaluation.  They  were  supplemented  by 
the  observations  and  recorded  comments  of  the  data  collector.  Instructor 
grades  were  collected  by  copying  the  Instructor's  grade  sheet,  including 
Instructor  comments.  ISS-generated  grades  were  either  copied  directly 
from  the  ISS  display  or  accessed  as  hardcopy  printout. 

RESULTS 

This  section  presents  the  results  of  the  comparison  of  Device  2F95 
training  conducted  with  and  without  the  assistance  of  the  ISS.  For  purposes 
of  comparison,  data  were  collected  for  a  total  of  50  trainer  events  for 
students  trained  with  the  ISS  and  54  trainer  events  for  students  trained 
with  the  standard  IOS.  In  addition,  data  were  collected  for  29  FAPT  050 
and  CQPT  040  events  all  of  which  were  trained  with  the  IOS.  A  summary  of 
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the  nuober  and  type  of  trainer  events  for  which  data  were  collected  Is  pro¬ 
vided  in  Table  1.  The  nuaber  of  events  shown  in  Table  1  provide  the  basis 
for  all  subsequent  data  collections. 

The  results  of  the  evaluation,  which  are  discussed  below,  were  arranged 
according  to  the  following  categories  of  measurement:  training  effectiveness, 
training  efficiency,  standardization  of  training,  effectiveness  and  utili¬ 
zation  of  special  instructional  features,  and  general  measures  of  ISS/IOS 
performance  and  operation. 

As  indicated  previously,  no  statistical  analyses  were  performed  on  the 
collected  data.  Therefore,  any  reported  differences  in  the  data  does  not 
necessarily  imply  either  statistical  significance  or  reliability. 

TRAINING  EFFECTIVENESS 

The  overall  effectiveness  of  ISS  training  was  assessed  by  comparing 
the  performance  of  students  trained  with  the  ISS  with  the  performance  of 
students  trained  with  the  IOS.  Within  the  context  of  this  study  the  only 
available  measures  of  student  performance  were  instructor  grades  for  simu¬ 
lator  training.  The  ISS  was  not  ready  to  train  all  of  the  FAM  events  or 
all  of  the  CQ  events.  Consequently,  FAM  and  CQ  training  for  students 
assigned  to  the  experimental  group  consisted  of  a  mix  of  events,  some  of 
which  were  trained  with  ISS  and  some  of  which  were  trained  with  the  IOS. 
Therefore,  it  would  have  been  impossible  to  establish  a  cause-effect  rela¬ 
tionship  between  mode  of  simulator  training  and  subsequent  flight  grades. 
In  addition,  CQ  events  CQPT  010,  CQPT  020  and  CQPT  030,  were  student- 
instructed  events,  hence  there  were  no  instructor  grades  available  for 
these  events. 

Table  2  shows  average  instructor  grades  for  FAM  events  FAPT  10,  FAPT 
020,  FAPT  030  and  FAPT  040  for  students  trained  with  the  ISS  and  students 
trained  with  the  IOS.  As  shown,  there  was  little  difference  in  instructor 
grades  for  individual  FAM  events  for  students  trained  with  the  ISS  or  the 
IOS.  The  overall  average  grade  received  for  FAM  training  event  was  3.13 
for  both  IOS  and  ISS  trained  students. 

Table  2  also  shows  grades  for  events  FAPT  050  and  CQPT  040.  Both  of 
these  events  were  trained  using  the  IOS.  However,  they  were  the  final 
simulator  training  events  that  just  preceded  the  first  aircraft  flight  for 
their  respective  stages  of  training.  As  such  they  provided  a  possible 
measure  of  the  effectiveness  of  the  previous  simulator  training  events  for 
that  stage.  Average  grades  for  the  two  groups  of  students,  however,  were 
essentially  identical.  The  average  FAPT  050  grade  was  3.14  for  IOS-trained 
students  and  3.15  for  ISS-trained  students.  Similarly,  the  average  CQPT 
040  grades  were  3.06  and  3.04,  respectively,  for  IOS  and  ISS  trained  stu¬ 
dents. 
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TABLE  1.  NUMBER  OF  TRAINING  EVENTS  FOR  WHICH  DATA 
COLLECTION  WAS  COMPLETED 


10 S -TRAINED  (CONTROL  GROUP) 


EVENT 

CAT  I 

CAT  II  &  III 

TOTAL 

FAPT  010 

5 

1 

6 

FAPT  020 

6 

1 

7 

FAPT  030 

6 

1 

7 

FAPT  040 

5 

1 

6 

TOTAL 

22 

4 

26 

CQPT  010 

3 

3 

6 

CQPT  020 

7 

4 

11 

CQPT  030 

7 

4 

11 

TOTAL 

17 

11 

28 

FAPT  050 

8 

1 

9 

CQPT 

11 

9 

20 

ISS-TRAINED  (EXPERIMENTAL  GROUP) 


EVENT 

CAT  I 

CAT  II  &  III 

TOTAL 

FAPT  010 

4 

0 

4 

FAPT  020 

6 

0 

6 

FAPT  030 

6 

0 

6 

FAPT  040 

6 

0 

6 

TOTAL 

22 

0 

22 

CAPT  010 

4 

2 

6 

CAPT  020 

7 

4 

11 

CAPT  030 

7 

4 

11 

TOTAL 

18 

10 

28 
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TABLE  2.  INSTRUCTOR  GRADES  OF  STUDENT  PERFORMANCE 


IOS -TRAINED  (CONTROL  GROUP) 


EVENT 

CAT  I 

CAT  II  &  III 

FAPT  010 

3.13 

3.25 

FAPT  020 

3.14 

3.07 

FAPT  030 

3.14 

3.13 

FAPT  040 

3.10 

3.06 

AVERAGE 

3.13 

3.13 

FAPT  050 

315 

3.13 

CQPT  040 

3.05 

3.07 

ISS-TRAINED  (EXPERIMENTAL  GROUP) 


EVENT 

CAT  I 

CAT  II  &  III 

FAPT  010 

3.23 

- 

FAPT  020 

3.09 

- 

FAPT  030 

3.11 

- 

FAPT  040 

3.10 

- 

AVERAGE 

3.13 

FAPT  050* 

3.15 

- 

CQPT  040* 

3.00 

3.15 

^Trained  with  the  IOS 
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TOTAL 

3.15 

3.13 

3.14 
3.09 

3.13 

3.14 
3.06 


TOTAL 

3.23 

3.09 

3.11 

3.10 

3.13 

3.15 

3.04 
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TRAINING  EFFICIENCY 

Training  efficiency  refers  to  the  amount  of  time  required  to  conduct 
training  activities  or  activities  that  are  required  to  support  training. 
Several  measures  of  efficiency  were  collected.  They  included:  (a)  the 
amount  of  time  required  to  set  up  (prepare)  the  training  device  for  a 
particular  event,  (b)  the  amount  of  time  required  to  reposition  the 
trainer  either  to  repeat  a  training  activity  or  to  initiate  another 
training  activity,  (c)  the  actual  amount  of  time  the  trainer  was  used  to 
conduct  training,  and  (d)  the  number  of  training  activities  or  events 
completed  per  training  session. 

TRAINER  SET-UP  TIMES.  The  amount  of  time  required  to  set-up  Device  2F95 
for  a  training  event  is  shown  in  Table  3.  Set-up  times  for  ISS  training 
were  uniformly  longer  than  set-up  times  for  IOS  training.  The  average 
trainer  set-up  time  for  FAM  events  was  1.72  minutes  for  IOS  training  and 
2.11  minutes  for  ISS  training.  Larger  differences  were  recorded  for  CQ 
training  where  set-up  times  averaged  1.30  and  3.96  minutes,  respectively, 
for  IOS  and  ISS  training. 


TABLE  3.  TRAINER  SET-UP  TIME  IN  MINUTES 


EVENT 

IOS  TRAINING 

ISS  TRAINING 

FAPT  010 

1.87 

2.75 

FAPT  020 

1.86 

2.58 

FAPT  030 

1.57 

1.83 

FAPT  040 

1.67 

1.50 

AVERAGE* 

1.72 

2.11 

CQPT  010 

1.50 

5.80 

CQPT  020 

1.27 

3.36 

CQPT  030 

1.21 

3.55 

AVERAGE 

1.30 

3.96 

*Note  that  numbers  of  training  events,  as  presented  in  Table  1, 
were  used  in  the  calulation  of  averages. 


The  longer  set-up  times  for  ISS  training  appeared  to  be  due  to  two 
factors.  First,  the  ISS  must  be  initialized  for  each  student,  whereas  the 
IOS  does  not  require  reinitialization  if  successive  students  are  being 
trained  on  the  same  event.  Second,  when  the  ISS  was  operated  in  the  STT 
mode,  as  it  was  for  CQ  training,  the  student  builds  his  own  training  sce¬ 
nario  by  selecting  from  a  menu  of  training  modules  for  the  stage  of  train¬ 
ing  he  was  interested  in.  This  required  some  planning  on  the  part  of  the 
Instructor  or  student  Inasmuch  as  the  entire  training  session  was  planned 
and  programmed  before  the  session  began.  The  ISS  then  automatically  sequenced 
the  student  through  the  selected  set  of  training  actlvties.  Students  using 
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the  IOS  tended  to  plan  only  one  event  at  a  time  which  initially  required 
little  or  no  set  up  time.  Then  they  either  repeated  the  same  event  over  or 
they  had  the  TD  reposition  them  for  another  event. 

Set-up  times  for  the  ISS  differed  depending  on  the  mode  in  which  it  was 
operated.  The  STT  mode  requires  the  most  set-up  time,  the  CANNED  mode  the 
least  time,  and  the  ISEL  mode  an  Intermediate  amount  of  set-up  time.  This 
difference  is  reflected  in  the  data  set-up  times  for  ISS  training;  set-up 
times  were  nearly  twice  as  long  for  CQ  events,  where  the  STT  mode  was  used, 
as  they  were  for  FAM  events  where  the  CANNED  and  ISEL  modes  were  used. 
There  also  was  some  evidence  that  both  students  and  instructors  learned  to 
accommodate  to  the  different  requirements  for  ISS  training  inasmuch  as  ISS 
set-up  times  decreased  over  successive  ISS  training  sessions. 

RESET/REPOSITION  TIMES.  There  were  no  differences  in  the  IOS  and  the  ISS 
in  the  amount  of  time  required  to  reset  the  trainer  to  repeat  a  given  train¬ 
ing  activity  or  to  reposition  it  for  another  activity.  However,  reset  and 
repositon  were  accomplished  automatically  by  the  ISS,  but  they  were  nor¬ 
mally  performed  manually  by  the  TD  at  the  instructor/operator  station  when 
the  IOS  was  used. 

ACTUAL  TRAINING  TIME  AND  TRAINING  ACTIVITIES.  The  amount  of  actual  training 
time,  exclusive  of  briefing  and  set-up  times,  was  calculated  for  IOS  and 
ISS  training.  However,  training  sessions  were  scheduled  for  a  fixed  amount 
of  time  and  therefore  were  influenced  by  the  amount  of  time  spent  briefing 
and  setting-up  the  trainer.  For  FAM  training,  where  differences  in  brief¬ 
ing  and  set-up  times  were  relatively  small,  the  average  amount  of  actual 
training  received  was  essentially  the  same  for  IOS  (87.1  minutes)  and  ISS 
(86.2  minutes)  training  events.  For  CQ  training,  where  there  was  a  larger 
difference  in  the  amount  of  time  spent  for  briefing  and  set-up,  the 
average  amount  of  actual  training  was  49.7  minutes  when  the  IOS  was  used 
compared  to  34.0  minutes  when  the  ISS  was  used.  The  average  number  of 
training  activities  (e.g.,  ILS  approach)  completed  per  CQ  training  session 
was  11.5  for  IOS  training  and  13  for  ISS  training.  However,  these  numbers 
are  not  directly  comparable  because  the  specific  manner  in  which  events 
were  trained  varied  between  IOS  and  ISS  controlled  training.  For  example, 
the  distance  from  the  carrier  to  the  initiation  point  of  an  ILS  approach 
varied  between  trials  and  between  students  when  the  IOS  was  used,  but  was 
always  the  same  when  the  ISS  was  used. 

STANDARDIZATION  OF  TRAINING 

SYLLABUS  DEVIATIONS.  A  potential  advantage  of  an  automated  instruction 
rupport  station  is  increased  standardization  of  training.  In  this  reference 
standardization  was  measured  by  counting  the  number  of  syllabus  deviations 
that  occurred  for  each  training  event.  A  syllabus  deviation  was  defined  as 
an  omission  of  a  training  activity  called  for  by  the  Trainer  Briefing  Guide 
(these  activities  are  listed  in  Appendix  A).  Syllabus  deviations  due  to 
equipment  malfunction  were  not  counted  nor  were  events  that  were  added  to 
the  syllabus. 
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Table  4  shows  the  average  number  of  syllabus  deviations  recorded  for 
each  FAM  and  CQ  training  event.  Overall,  the  average  number  of  syllabus 
deviations  per  training  session  was  slightly  higher  for  FAM  training  when 
the  10S  was  used  (2.42)  than  when  the  ISS  was  used  (2.09).  On  the  other 
hand,  the  number  of  syllabus  deviations  was  slightly  lower  for  CQ  training 
when  the  IOS  was  used  (1.78)  than  when  the  ISS  was  used  (2.00).  The  dif¬ 
ference  between  FAM  and  CQ  training  probably  were  due  to  the  mode  in  which 
the  ISS  was  operated.  That  Is,  the  ISS  was  operated  in  the  CANNED  and 
ISEL  modes  for  FAM  training  which  provided  less  flexibility  in  selecting 
training  activities.  However,  the  ISS  was  operated  in  the  STT  Mode  for  CQ 
training.  This  mode  provided  more  flexibility  in  selecting  syllabus  events 
and  hence  may  have  resulted  in  more  syllabus  deviations. 

The  most  frequent  syllabus  deviations  and  their  relative  frequency  of 
occurrence  are  shown  in  Table  5  for  FAM  training  and  in  Table  6  for  CQ 
training. 

AUTOMATED  PERFORMANCE  MEASUREMENT  AND  GRADING.  The  automated  performance 
measurement  and  grading  capability  of  the  ISS  affords  an  opportunity  to 
increase  the  standardization  of  instructor-assigned  grades  of  student  per¬ 
formance.  However,  this  feature  of  the  ISS  was  the  subject  of  a  separate 
contract  and  will  not  be  discussed  here  except  with  reference  to  its  poten¬ 
tial  contribution  as  a  special  instructional  feature. 

EFFECTIVENESS  AND  UTILIZATION  OF  ISS  INSTRUCTION  FEATURES 

An  attempt  was  made  to  assess  the  effectiveness  and  utilizaton  of  ISS 
instructional  features  that  are  not  currently  provided  by  the  IOS.  Some  of 
these  functions  are  especially  well  suited  to  assist  the  student  and  in¬ 
structor  during  briefing  and  debriefing  activities.  Other  ISS  functions  are 
not  specific  to  a  particular  training  activity  and  are  therefore  addressed 
simply  as  special  instructional  features. 

AUTOMATED  BRIEFING.  An  important  aspect  of  training  involves  adequately 
preparing  the  student  for  the  training  he  is  to  receive.  This  may  involve 
ensuring  that  he  has  sufficient  knowledge  to  benefit  from  training,  explain¬ 
ing  the  content  or  course  of  training,  providing  special  instructions,  high¬ 
lighting  things  to  look  for,  reviewing  procedures,  etc.  Special  features 
of  the  ISS  would  allow  a  student  to  brief  himself  or  they  could  be  used  as 
an  aid  by  the  instructor  to  conduct  the  briefing. 

The  original  training  concept  was  to  provide  a  remote  (secondary)  ISS 
instructor’s  console  that  could  be  used  off-line  by  the  student  or  instruc¬ 
tor  prior  to  the  scheduled  simulator  training  event.  However,  the  remote 
console  was  not  operating  during  the  data  collection  period.  Therefore, 
the  automated  briefing  function  of  the  ISS  had  to  be  utilized  from  the 
primary  Instructor's  console  that  was  being  used  to  conduct  training.  In 
most  cases  the  trainers  were  scheduled  such  that  there  was  no  free  time 
available  between  successive  trainer  events.  Consequently,  instructors 
and  students  rarely  had  an  opportunity  to  use  the  ISS  before  their  sched¬ 
uled  training  session.  Students  and  instructors  usually  completed  their 
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TABLE  4.  SYLLABUS  DEVIATIONS  PER  TRAINING  SESSION 


IOS-TRAINING  (CONTROL  GROUP) 


EVENT 

MEAN  NUMBER  OF  : 

FAPT  010 

4.50 

FAPT  020 

1.14 

FAPT  030 

2.29 

FAPT  040 

2.00 

AVERAGE* 

2.42 

CAPT  010 

1.83 

CAPT  020 

1.55 

CAPT  030 

2.00 

AVERAGE 

1.78 

ISS-TRAINING  (EXPERIMENTAL  GROUP) 


EVENT 

MEAN  NUMBER  OF  1 

FAPT  010 

4.00 

FAPT  020 

2.33 

FAPT  030 

1.00 

FAPT  040 

1.67 

AVERAGE 

CAPT  010 

1.83 

CAPT  020 

1.91 

CAPT  030 

2.18 

AVERAGE 

2.00 

*Note  that  numbers  of  training  events,  as  presented  In  Table 
1,  were  used  In  the  calculation  of  averages. 
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TABLE  5.  MOST  FREQUENT  SYLLABUS  DEVIATIONS  FOR  FAM  TRAINING 


MODE  OF  PERCENTAGE  OF 

TRAINING  EVENT  SYLLABUS  DEVIATION/OMISSION  TOTAL  DEVIATIONS 


IOS 

FAPT  010 

1.  CIRCUIT  BREAKER  RE  #1  FAIL 

17.9% 

2.  SPOILER  LIGHT 

10.7% 

3.  MAC  TRIM  ADVISORY  LIGHT 

10.  7% 

ISS 

FAPT  010 

1.  CIRCUIT  BREAKER  RE  # 1  FAIL 

18.9% 

2.  RIGHT  ENGINE  FUEL  PRESSURE  FAIL 

12.5% 

3.  EMERGENCY  GENERATOR  FAIL 

12.5% 

4.  SINGLE-ENGINE  APPROACH 

12.5% 

IOS 

FAPT  020 

1.  NO  CANOPY  LIGHT 

2.  NO  WAVE-OFF  ON  FIRST  SINGLE¬ 

25% 

ENGINE  APPROACH 

25% 

ISS 

FAPT  020 

1.  NO  WILD  CARD  MALFUNCTION 

2.  REMOVED  UNSCHEDULED  WING-SWEEP 

31.3% 

MALFUNCTION  BEFORE  INSERTING 

SPOILER  LIGHT 

25% 

IOS 

FAPT  030 

1.  NO  WILD  CARD  MALFUNCTION 

2.  RIGHT  AICS  FAILURE  AND  RIGHT 

31.3% 

RAMP  LIGHT  GIVEN  SEPARATELY 

INSTEAD  OF  TOGETHER 

31.3% 

3.  LEFT  EXHAUST  NOZZLE  FAILURE  AND 

OVERSPEED  VALVE  LIGHT  GIVEN 
SEPARATELY  INSTEAD  OF  TOGETHE 

12.5% 

4.  BLOWN  TIRE 

12.5% 

ISS 

FAPT  030 

1.  NO  WILD  CARD  MALFUNCTION 

37.5% 

2.  BLOWN  TIRE 

37.5% 

IOS 

FAPT  040 

1.  BLOWN  TIRE  ON  TAKEOFF 

25% 

2.  CANOPY  LIGHT 

16.7% 

3.  AIRSPEED  INDICATOR  FAIL 

16.7% 

ISS 

FAPT  040 

1.  NO  WILD  CARD  MALFUNCTION 

37.5% 

2.  FLAP  MALFUNCTION 

25% 

33 


NAVTRAEQUIPC EN-8 1 -C-008 1 - 1 


TABLE  6.  MOST  FREQUENT  SYLLABUS  DEVIATIONS  FOR  CQ  TRAINING 


MODE  OF 

PERCENTAGE  OF 

TRAINING 

EVENT 

SYLLABUS  DEVIATION/OMISSION 

TOTAL  DEVIATIONS 

IOS 

CQPT  010 

1.  NO 

BOX  PATTERN 

45.5% 

2.  NO 

MIRAMAR  LANDINGS 

45.5% 

ISS 

CQPT  010 

1.  NO 

BOX  PATTERN 

36.4% 

2.  NO 

MIRAMAR  LANDINGS 

36.4% 

3.  NO 

TAKEOFF 

27.3% 

IOS 

CQPT  020 

1.  NO 

MARSHAL  PATTERN 

41.2% 

2.  NO 

BOLTER  PATTERN 

29.4  % 

3.  NO 

CV  1  APPROACH 

23.5% 

ISS 

CQPT  020 

1.  NO 

MARSHAL  PATTERN 

52.4% 

2.  NO 

CV  1  APPROACH 

19.0% 

IOS 

CQPT  030 

1.  NO 

BINGO  PROFILE 

45.5% 

2.  NO 

CV  3  APPROACH 

27.3% 

3.  NO 

OVERHEAD  MARSHAL  PATTERN 

27.3% 

ISS 

CQPT  030 

1.  NO 

BINGO  PROFILE 

45.8% 

2.  NO 

OVERHEAD  MARSHAL  PATTERN 

29.2  % 

3.  NO 

CV  3  APPROACH 

25.0% 
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briefings  just  before  their  scheduled  training  in  order  to  fully  utilize 
the  time  they  had  available  for  simulator  training.  They  were  reluctant, 
therefore,  to  spend  simulator  training  time  to  use  the  ISS  for  briefing. 
In  addition,  a  brief  demonstration  of  the  capabilty  of  the  ISS,  and  on- 
the-job  training  on  how  to  use  it,  were  given  by  Logicon  personnel  for  the 
first  one  or  two  FAM  and  CQ  events,  except  for  those  cases  during  FAM 
training  where  the  instructor  already  had  been  trained  on  the  ISS.  This 
training  was  given  at  the  beginning  of  the  training  session  and  had  the 
effect  of  reducing  the  amount  of  time  available  to  conduct  simulator 
training. 

The  unavailability  of  a  remote  ISS  instructor's  console  and  the  conduct 
of  on-the-job  training  prevented  an  accurate  assessment  of  the  extent  to 
which  the  automated  briefing  function  might  have  been  used  if  a  remote 
instructor's  console  had  been  available  or  of  the  utility  of  that  feature. 
However,  briefing  times  were  collected  for  both  IOS  and  ISS  training.  For 
CQ  training  there  were  no  instructor  briefings.  The  ISS  was  used  for  brief¬ 
ing  ISS-controlled  events,  however  this  time  usually  consisted  of  demon¬ 
stration  and  training  and  there  was  no  way  of  assessing  how  much  the  student 
would  have  used  the  ISS  for  briefing  purposes  under  normal  circumstances. 
Briefing  times  were  slightly  longer  for  FAM  training  when  the  ISS  was  used, 
however  this  longer  briefing  time  again  was  due  primarily  to  ISS  demon¬ 
stration  and  training. 

AUTOMATED  DEBRIEFING.  The  ISS  provided  a  record/playback  feature  and  an 
automated  performance  measurement  and  grading  capability  that  could  be 
used  by  a  student  to  debrief  himself  or  as  an  aid  to  the  instructor  during 
debriefing.  However,  the  unavailability  of  a  remote  instructor's  console 
made  it  difficult  for  students  or  instructors  to  use  these  features. 
Normally,  by  the  time  simulator  training  had  been  completed  there  was  little 
or  no  time  remaining  before  the  next  scheduled  training  session.  In 
addition,  the  debriefing  capability  of  the  ISS  usually  was  demonstrated  by 
Logicon  personnel  during  this  time. 

Debriefing  times  were  collected  for  all  FAM  and  CQ  events.  Debriefing 
times  averaged  12.0  minutes  for  FAM  training  conducted  with  the  IOS  and 
17.5  minutes  when  training  was  conducted  using  the  ISS.  However  the  ISS 
Itself  was  used,  on  the  average,  only  1.5  minutes  per  debriefing.  For  CQ 
training  there  were  no  debriefings  for  IOS-trained  students  because  CQ 
events  were  self-instructed.  Debriefing  was  available  for  ISS-trained 
students  who  used  the  ISS  an  average  of  5.2  minutes  for  this  purpose,  how¬ 
ever,  this  measure  of  ISS  utilization  includes  the  time  taken  to  demonstrate 
its  debriefing  capability.  Typically,  there  was  more  time  available  at 
the  end  of  CQ  training  events  than  there  was  for  FAM  events.  In  addition, 
CQ  students  appeared  to  be  interested  in  reviewing  their  performance  and 
in  obtaining  some  feedback  concerning  that  performance,  especially  for 
carrier  approaches  and  landings. 

SPECIAL  INSTRUCTION  FEATURES.  The  ISS  includes  special  instructional  fea¬ 
tures  in  addition  to  those  already  discussed.  Some  of  these  are:  computer¬ 
generated  speech,  automated  checklists  and  procedures  monitoring,  reposition 
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capability,  special  graphics  display,  plus  several  special  control  features 
that  can  be  effected  from  the  ISS  console  (e.g.  changes  in  environmental 
conditions,  instrument  failures,  GCA  wave-off  and  freeze). 

An  attempt  was  made  to  assess  how  often  these  special  features  were 
used,  when  utilization  was  an  instructor's  option,  and  to  provide  an  esti¬ 
mate  of  the  effectiveness  or  utility  of  these  features  from  an  istruc- 
tional  standpoint. 

Table  7  shows  the  number  of  times  per  training  session  that  instruc¬ 
tional  features  were  used  or  selected  by  the  instructor.  The  data  of  Table 
7  are  for  FAM  training  only;  CQ  events  were  not  conducted  by  instructors. 

The  instructional  features  listed  in  Table  7,  plus  several  features 
that  were  part  of  the  ISS  but  not  instructor  options,  are  discussed  briefly 
below. 

Automated  Checklist.  When  this  feature  was  selected,  the  checklist,  as 
depicted  in  the  NATOPS  manual,  was  shown  on  the  lower  display  of  the  ISS 
console.  The  pilot's  timed  actions  was  indicated  for  those  items  that  are 
measurable  by  the  ISS.  For  those  items  not  measurable,  the  instructor 
inserted  his  evaluation  of  student  performance  via  "satisfactory  and 
"unsatisfactory"  touch  panels  also  shown  on  the  display.  For  FAM  Class 
8210  the  checklist  format  was  changed.  In  addition  to  being  displayed  on 
the  ISS  console  the  checklist  was  also  given  by  the  ISS  automated  voice- 
generation  system.  However,  there  was  no  option  to  have  the  display 
only. 


The  automated  checklist  is  convenient  because  it  eliminates  the  neces¬ 
sity  of  having  a  copy  of  the  appropriate  checklist  available.  It  also  is 
useful  for  monitoring  pilot  actions  that  are  measurable  during  checklist 
procedures.  Pilot  performance  of  checklist  procedures  also  is  automati¬ 
cally  recorded  and  can  be  used  for  subsequent  debriefing  purposes. 

Automated  checklists  were  used  with  some  regularity,  however  the  auto¬ 
mated  voice  generated  checklist  was  not  well  received.  Students  complained 
that  the  automated  voice  sequenced  through  the  checklist  too  quickly,  and 
that  sometimes  it  was  hard  to  understand.  Moreover  radar  intercept  offi¬ 
cers  (RIO),  who  are  normally  present  during  pilot  training,  preferred  to 
have  the  option  of  making  checklist  calls  themselves. 

Computer-Generated  Voice  Transmissions.  Voice  transmissions  which  pre¬ 
viously  were  given  by  the  instructor  to  simulate  the  various  controlling 
agencies  were  automatically  transmitted  to  the  cockpit  via  the  ISS  voice 
generation  unit.  The  simulated  agencies  Included  the  tower  for  take-off 
clearances,  departure  instructions,  approach  control  for  radar  contact/ 
approach  clearances  and  radar  vector  instructions,  and  final  conroller 
for  CCA/GCA  instructions  and  bolter/missed-approach  instructions. 


This  feature  was  used  frequently  and  was  well  received  by  both  students 
and  instructors.  It  relieved  the  instructor  from  the  task  of  making  tower 
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TABLE  7.  AVERAGE  NUMBER  OF  TIMES  ISS  SPECIAL  FEATURES 
USED  PER  TRAINING  SESSION 


NUMBER  OF  TIMES 

TYPE  OF  SPECIAL  FEATURE  USED  PER  SESSION 

AUTOMATED  TAKE-OFF  CHECKLIST  (DISPLAY)  0.83 

AUTOMATED  ASCENT  CHECKLIST  (DISPLAY)  0.33 

AUTOMATED  DESCENT  CHECKLIST  (DISPLAY)  1.00 

AUTOMATED  LANDING  CHECKLIST  (DISPLAY)  0.67 

AUTOMATED  TAKE-OFF  CHECKLIST  (VOICE)  0.00 

AUTOMATED  ASCENT  CHECKLIST  (VOICE)  0.13 

AUTOMATED  DESCENT  CHECKLIST  (VOICE)  0.13 

AUTOMATED  LANDING  CHECKLIST  (VOICE)  0.00 

COMPUTER-GENERATED  VOICE  COMMUNICATIONS*  0.50 

TRAINER  REPOSITON  1.05 

ENVIR01MENT  OPTIONS  0.32 

INSTRUMENT  FAILURE  0.32 

AUTOMATED  GCA  WAVE-OFF  0.09 

TRAINER  FREEZE  0.18 


*This  refers  only  to  the  number  of  times  the  automated  voice  communica¬ 
tions  was  selected  to  give  clearances.  Computer-generated  voice  communi¬ 
cations  were  always  given  for  final  approaches. 
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and  approach  communications;  It  also  Improved  the  standardization  of  those 
communications.  The  automated  voice  was  especially  useful  for  CQ  training 
because  there  was  no  instructor  available  to  simulate  the  normal  voice 
communications  for  those  events.  Instructors  also  liked  the  fact  that 
automated  voice  communications  continued  during  simulated  emergency  con¬ 
ditions.  This  forced  the  pilot  and  RIO  to  contend  with  outside  communica¬ 
tions  while  coordinating  their  own  activities  and  communication,  which  is 
similar  to  the  situation  they  will  likely  encounter  in  actual  fight 
exercises. 

The  only  notable  complaint  about  the  computer-generated  voice  was  that 
it  occasionally  was  hard  to  understand. 

Trainer  Repositioning.  This  feature  allowed  the  instructor  to  reposition 
the  trainer  to  selected  preprogrammed  locations.  The  instructor  had  the 
option  of  selecting  radar  vectors  to  these  locations. 

This  feature  was  used  frequently  and  it  effectively  allowed  the  in¬ 
structor  to  alter  the  structure  of  training,  or  to  eliminate  enroute  flight 
segments.  The  ISS  repositioning  feature  was  less  flexible  than  the  aircraft 
slewing  feature  provided  by  the  IOS.  A  limited  set  of  repositioning  coor¬ 
dinates  were  available  on  the  ISS  and  these  fixed  coordinates  included  a 
predetermined  altitude  setting.  On  the  other  hand,  the  slewing  feature  of 
the  IOS  allowed  the  aircraft  to  be  repositioned  to  any  desired  set  of 
coordinates  or  altitude  settings,  although  this  feature  is  more  difficult 
and  time  consuming  to  use. 

Environmental  Options.  This  feature  provides  the  instructor  with  the 
option  of  changing  environmental  conditions  (e.g.  wind  speed,  wind  direc¬ 
tion,  and  air  turbulence)  and  hence  allowed  him  to  increase  the  difficulty 
of  flying  the  aircraft.  The  feature  was  occasionally  selected  by  instruc¬ 
tors,  however,  the  instructional  intent  of  selecting  environmental  options 
usually  was  unclear. 

Instrument  Failure.  This  feature  allowed  the  instructor  to  fail  the  air¬ 
craft  automatic  throttle,  ILS,  or  compass  from  the  ISS  console.  It  afforded 
the  instructor  some  flexibility  in  altering  the  difficulty  of  flying  the 
aircraft,  especially  during  landing  approaches. 

Automated  GCA  Wave-off.  This  feature,  when  selected,  automatically  gave  a 
wave-off  call  and  missed  approach  procedures  whenever  aircraft  approach 
exceeded  established  glide  slope  and  azimuth  parameters  or  whenever  the 
Instructor  selected  a  wave-off  call  that  required  the  student  to  practice 
missed-approach  procedures.  This  feature  functioned  effectively,  although 
it  was  not  used  very  often. 

Graphics  Display.  This  feature  provided  graphic  information  on  the  upper 
ISS  console  display  concerning  (a)  aircraft  position  relative  to  geographic 
positions  in  the  form  of  operating  areas,  airways,  standard  instrument 
approaches  and  departures  and  (b)  final  approaches  wih  respect  to  glide 
slope  and  azimuth.  This  latter  display  showed  a  historical  trail  of 
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aircraft  flight  path  with  respect  to  desired  glide  slope  and  heading. 
These  graphics  displays  occurred  automatically  and  were  not  selected  by 
the  Instructor  or  operator. 

The  graphics  display  appeared  to  be  a  useful  feature  especially  for 
diagnoaing  or  detecting  landing  approach  or  bolter  pattern  error.  The 
historical  trail  of  aircraft  flight  path  allowed  the  Instructor  to  see  the 
overall  pattern  of  the  approach  and  to  easily  identify  consistent  errors 
revealed  by  the  similarity  of  these  patterns. 

The  graphics  display  is  Included  in  the  1SS  playback  feature  and 
potentially  could  be  a  valuable  debriefing  aid,  especially  for  situations 
where  an  Instructor  is  not  available. 

GENERAL  MEASURE  OF  ISS/IOS  PERFORMANCE  AND  OPERATION 

Two  measures  were  collected  to  assess  the  overall  operation  and  per¬ 
formance  of  the  ISS  and  the  IOS.  First,  data  were  collected  on  the  ease 
of  operating  the  respective  instructor/operator  consoles,  as  measured  by 
the  number  of  errors  made  by  Instructors/operators  during  a  normal  train¬ 
ing  session.  Second,  data  were  collected  on  the  frequency  of  equipment 
malfunctions  which  Interrupted  or  interfered  with  training. 

INSTRUCTOR/OPERATOR  ERRORS.  Anytime  humans  are  required  to  operate  or 
interact  with  sophisticated  hardware,  errors  are  likely  to  occur  that  are 
attributable  to  the  human  operator.  The  number  and  type  of  instructor/ 
operator  errors  were  recorded  for  training  conducted  with  the  ISS  and  with 
the  IOS.  Errors  were  defined  as  omitted  actions,  incorrect  actions, 
actions  performed  out  of  sequence,  or  actions  performed  at  the  wrong  time 
which  were  related  to  the  operation  of  the  instructor's  console  and  which 
had  a  noticeable  effect  on  training. 

During  FAM  training  an  average  of  1.85  instructor/operator  errors  were 
recorded  per  training  event  when  training  was  given  from  the  IOS.  However, 
only  0.50  errors  per  training  event  were  recorded  when  training  was  given 
from  the  ISS.  Similarly,  1.5  errors  per  training  event  were  recorded  for 
CQ  training  when  the  IOS  was  used,  but  essentially  no  errors  were  recorded 
when  the  ISS  was  used.  The  lack  of  errors  for  CQ  training  was  due  to  the 
fact  that  the  ISS  did  not  require  an  instructor  or  operator  for  these 
events. 

Table  8  shows  the  most  frequently  committed  errors  and  the  relative 
frequency  with  which  they  occurred.  For  FAM  training  the  most  frequently 
committed  error  involved  inserting  the  wrong  malfunction,  regardless  of 
whether  the  training  was  conducted  with  the  IOS  or  with  the  ISS.  However, 
other  operator  errors  that  occurred  when  the  IOS  was  used  were  essentially 
eliminated  when  the  ISS  was  used.  A  similar  finding  was  obtained  for  CQ 
training;  errors  attributed  to  the  IOS  operator  did  not  occur  when  the  ISS 
was  used. 

EQUIPMENT  MALFUNCTIONS.  Equipment  malfunctions  that  interrupt  or  inter¬ 
fere  with  the  smooth  conduct  of  training  can  have  deleterious  effects. 
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MODE  OF 
TRAINING 

IOS 

ISS 


MODE  OF 
TRAINING 

IOS 

ISS 


TABLE  8.  TYPE /SOURCE  AND  RELATIVE  FREQUENCY  OF 
INSTRUCTOR/OPERATOR  ERRORS 


FAM  TRAINING 


PERCENTAGE  OF 


TYPE /SOURCE  OF  ERROR  TOTAL  ERRORS 

1.  INSERTED  WRONG  MALFUNCTION  33.3% 

2.  DISCONNECTED  POWER/AIR  TO  AIRCRAFT  TOO  EARLY  12.5% 

3  FAILURE  TO  INITIATE  RUN  COMMAND  8.3% 

4  FAILURE  TO  REMOVE  WHEEL  CHOCKS  8-3% 

1.  INSERTED  WRONG  MALFUNCTION  54.5% 


CQ  TRAINING 

PERCENTAGE  OF 


TYPE /SOURCE  OF  ERROR  TOTAL  ERRORS 

1.  FAILURE  TO  INITIATE  RUN  COMMAND  14.2% 

2  NO  TRAINING  DEVICE  OPERATOR  (TD) 

AVAILABLE  WHEN  NEEDED  11.9% 

3.  INITIALIZATION  TO  WRONG  COORDINATES  11.9% 

4.  PROGRAMMED  WRONG  PARAMETER  11.9% 


N/A 
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Overall  training  effectiveness  can  be  decreased  by  breaking  the  continuity 
of  training  or  by  making  necessary  the  exclusion  of  some  aspect  of  training. 
In  addition,  equipment  malfunctions  that  occur  during  a  training  session 
can  be  frustrating  for  both  the  student  and  the  instructor.  Such  frustra¬ 
tion  can,  in  turn,  negatively  affect  user  attitudes  and  influence  subse¬ 
quent  utilization  of  the  trainer. 

The  number  of  equipment  (hardware  and  software)  malfunctions  that 
interrupted  or  interfered  with  training  were  recorded  for  IOS-  and  ISS- 
tralned  students.  Equipment  malfunctions  that  occurred  during  ISS  train¬ 
ing  were  categorized  according  to  whether  they  required  the  ISS  to  be 
reinitialized  or  whether  it  was  necessary  to  terminate  ISS  training  and 
complete  the  training  session  with  the  IOS. 

Overall,  out  of  50  ISS  training  sessions,  nine  had  to  be  finished  with 
the  IOS  because  of  malfunctions;  seven  during  FAM  training  and  two  during 
CQ  training.  An  additional  10  ISS  training  sessions  were  completed  with 
the  IOS  because  the  ISS  was  not  flexible  enough  to  accommodate  changes  in 
training  requested  or  desired  by  the  instructor. 

On  eight  occasions  it  was  necessary  to  reinitialize  the  ISS  due  to 
equipment  malfunctions.  This  normally  interrupted  training  for  one  or  two 
minutes,  although  a  modification  made  to  the  ISS  near  the  end  of  data 
collection  activities  allowed  reinitialization  to  occur  without  interrupt¬ 
ing  training.  The  ISS  was  reinitialized  an  additional  seven  times;  three 
times  because  the  Instructor  changed  his  mind  about  what  events  to  train, 
and  four  times  because  the  ISS  was  not  flexible  enough  to  accommodate 
changes  In  training  requested  by  the  instructor. 

Out  of  83  training  sessions  conducted  with  the  IOS,  including  FAPT  050 
and  CQPT  040,  equipment  malfunctions  interrupted  training  approximately  10 
times.  There  were  no  recorded  instances  where  training,  once  started  with 
the  IOS,  had  to  be  terminated.  Training  Interruptions  on  the  IOS  usually 
were  due  to  a  requirement  to  "boot  strap”  the  computer. 

The  relatively  high  "failure"  rate  recorded  for  the  ISS  is  somewhat 
misleading.  In  nearly  every  instance  the  requirement  to  terminate  ISS 
training  was  due  to  what  appeared  to  be  a  simple  software  error  that  often 
was  solved  by  the  next  scheduled  day  of  training.  Development  and  modifi¬ 
cation  of  the  ISS  software  continued  throughout  data  collection  activities. 
Therefore,  the  frequency  of  ISS  errors  recorded  may  perhaps  be  attributed 
to  the  fact  that  the  ISS  was  an  experimental  prototype  without  sufficient 
time  available  for  software  debugging,  rather  than  a  table  system  with 
inherent  failure  problems. 
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SECTION  V 


PERFORMANCE  MEASUREMENT  VIEW 


Performance  measurement  for  a  training  system  can  be  viewed  fron  two 
perspectives  as  a  major  element  in  a  data  processing  system,  and  as  a  source 
of  information  to  support  training  activities.  The  data  processing  view 
is  one  of  concern  for  data  format,  accuracy,  completeness  and  efficiency. 
The  information  generator  view  is  one  of  concern  for  the  adequacy  of  the 
information  to  provide  complete  and  understandable  information  for  training 
feedback  and  decision  requirements.  Each  of  these  views  will  be  addressed 
in  this  section. 

DATA  PROCESSING 

The  procedure  for  evaluating  the  F-14  ISS  performance  measurement, 
from  a  data  processing  point  of  view,  was  to  collect  an  adequate  sample  of 
data  and  perform  trial  processing.  More  specifically,  the  data  analysis 
objectives  were  to  (a)  improve  discrimination  between  different  levels  of 
performance,  (b)  improve  machine-derived  grades  to  increase  correlation 
with  those  which  would  be  determined  by  the  Instructors,  (c)  determine  how 
the  ISS  was  used  including  use  of  specific  design  features,  and  (d)  evaluate 
the  measurement  implementation  in  terms  of  adequacy  to  achieve  multiple 
uses  of  the  recorded  data.  Unfortunately,  as  commented  elsewhere  in  this 
report,  insufficient  data  were  collected  to  fully  accomplish  these  objec¬ 
tives,  however,  the  following  paragraphs  will  address  these  topics  to  the 
extent  possible. 

METHOD.  Computer  magnetic  tapes  (Standard  IBM  9-Track  Format)  and  associated 
listings  were  obtained  from  Logicon,  Incorporated  as  data  were  collected. 
There  were  two  types  of  data:  Audit  Trail  data,  indicating  the  use  of  system 
features,  and,  Session  data,  indicating  measures  of  student  performance. 
The  magnetic  tapes  were  read  on  a  DEC  VAX  at  the  Naval  Training  Equipment 
Center,  Orlando,  Florida,  and  after  development  of  appropriate  procedures 
these  proved  to  be  readable.  Data  were  also  transferred  to  8-inch  floppy 
disks  from  the  DEC  VAX,  and  then  subsequently  to  8-inch  floppy  disks  in 
CP/M  format  using  software  acquired  for  this  purpose.  It  was  Intended  that 
preliminary  analyses  would  be  conducted  on  a  Z-80  microcomputer  to  provide 
a  quick  examination  of  the  data;  however,  as  it  developed,  all  analyses 
were  done  on  the  microcomputer.  The  microcomputer  proved  to  be  surprisingly 
capable,  limited  only  by  the  amount  of  data  which  could  be  contained  on  the 
floppy  disks.  The  microcomputer  analysis  included  cross-checking  between 
measures  to  detect  outlying  data  points  (manual  checks  were  also  conducted) , 
regression  analyses  and  discriminant  analyses. 

MEASURE  DISCRIMINATION.  The  measurement  computed  in  the  ISS  must  be  capable 
of  discriminating  between  the  performance  of  inexpert  students,  adequately 
performing  students,  expert  performers,  and  those  with  performance  problems 
of  various  kinds.  A  multivariate  discriminant  analysis,  comparing  the 
performance  of  groups  of  subjects  for  each  important  performance  class, 
will  permit  identification  of  those  ISS  measures  (if  any)  which  can  pro¬ 
vide  the  desired  discriminations. 
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A  ridge-adjusted  dlscrioinant  analyses  was  progranuned  in  the  FORTRAN 
computer  language.  A  non-parametric  version  is  also  available  for  small 
samples.  These  analysis  tools  were  applied  to  samples  of  the  available 
data  as  a  test  of  data  quality,  however,  sufficient  quantities  of  data  did 
not  become  available  to  permit  fully  executing  these  analyses.  Nevertheless, 
based  on  the  data  examined,  data  quality  from  the  ISS  should  suffice  for 
these  purposes,  and  the  complete  analysis  can  be  conducted  when  additional 
data  collection  is  accomplished. 

CORREIATION  WITH  INSTRUCTOR  GRADES.  A  multivariate  regression  analysis 
was  developed  with  the  objective  of  identifying  key  measures  and  appro¬ 
priate  weightings  which  could  then  be  used  to  compute  scores  and  grades 
equivalent  to  those  produced  by  the  Instructors.  A  standard  regression 
analysis  and  a  ridge-adjusted  regression  analysis  were  programmed  in 
FORTRAN.  These  programs  were  tested  with  constructed  standard  data  and 
with  data  collected  on  the  F-14  ISS.  Since  the  only  data  collected  in  any 
quantity  were  landings  of  various  types,  these  were  collected  together  to 
form  a  composite  test  data  base  (a  total  of  43  landings). 

A  measurement  structure  was  used  in  the  ISS  wherein  a  parameter,  such 
as  Glideslope  or  Centerline  deviation,  was  divided  into  three  regions 
termed  ON,  SLIGHTLY  OFF,  and  WELL  OFF.  A  count  of  the  number  of  samples  of 
data  in  each  category  was  made  and  then  a  percentage  of  the  total  number 
of  samples  was  calculated.  The  regression  analysis  found  the  ON  and  WELL 
OFF  measures  to  be  negatively  correlated  (.84  -  .95)  as  could  be  expected 
(e-g.  pilots  who  were  not  on  course  were  therefore  off  course),  one  of  the 
two  measures  had  to  be  deleted  to  conduct  a  successful  regression  analysis. 
This  undesirable  redundancy  should  be  considered  in  any  redesign  of  ISS 
measurement. 

Unfortunately,  the  regression  analysis  could  not  achieve  the  intended 
goal  since  the  small  amount  of  data  collected  consisted  mainly  of  student- 
only  training,  without,  of  course,  any  instructor  grade  recorded.  Never¬ 
theless,  the  analysis  stands  in  readiness,  and  can  be  executed  when  suitable 
types  and  quantities  of  data  are  available. 

USE  OF  DESIGN  FEATURES.  A  FORTRAN  program  was  prepared  to  tabulate  the 
use  of  Identified  system  features  for  each  month  of  Audit  Trail  data.  As 
shown  in  Figure  4,  a  count  of  system  features  for  each  type  of  user  is 
tabulated,  and  the  total  number  of  hours  of  use  for  each  type  of  user  is 
calculated.  These  data  should  permit  identification  of  those  features  which 
are  heavily  and  lightly  used.  Given  adequate  quantities  of  data,  these 
should  provide  valuable  information  for  system  analysis  and  redesign. 

ADEQUACY  OF  DATA  GENERATED.  As  noted  above,  the  data  collected  were  sub¬ 
jected  to  a  variety  of  computer  analyses.  These  analyses  contained  checks 
on  the  range  of  values  and  relationships  between  parameters  when  possible. 
All  data  were  listed  and  manually  reviewed.  As  a  result  of  these  inspec¬ 
tions  numerous  errors  were  found;  however,  in  all  cases  these  were  either 
the  result  of  aborted  demonstration  sessions,  equipment  malfunctions,  or 
software  bugs.  All  errors  were  judged  to  be  correctable  and  data  error 
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USE  OF  ISS  FEATURES  SEP 


FEATURE 

INSTR 

STUDENT 

TEST 

OTHER 

LOGON 

27 

32 

143 

0 

I DENT  STUDENT 

10 

0 

13 

0 

ENTER  EX.  DEFIN. 

0 

24 

33 

0 

RE I NT  EX.  DEFIN. 

0 

3 

6 

0 

COMPL  EX.  DEFIN. 

0 

24 

29 

0 

BRIEF  MISSION 

13 

21 

89 

0 

ACTIV.  MALF 

20 

10 

14 

0 

REMOVE  MALF- 

16 

5 

2 

0 

SEL.  ENVIR.  OPT. 

20 

0 

9 

0 

REPOSITION  A/C 

9 

2 

5 

0 

EXAMINE  CHKLST 

25 

6 

10 

0 

REMOVE  CHKLST 

1 

0 

1 

0 

SEL.  MAN.  MODE 

4 

17 

30 

0 

FREEZE  SYS. 

0 

0 

7 

0 

REQU.  HARDCOPY 

1 

6 

74 

0 

REQU.  AUTOVOICE 

0 

0 

0 

0 

DEBRIEF  MISSION 

11 

33 

42 

0 

EXAMINE  CRITERIA 

0 

4 

13 

0 

PLAYBACK  MODULE 

4 

38 

3 

0 

CHANGE  GRADE 

0 

0 

5 

0 

LOGOFF 

22 

37 

182 

0 

REVIEW  SUMMARY 

1 

2 

7 

0 

REVIEW  EXPANDED 

1 

4 

16 

0 

VIEW  DISPLAY 

13 

9 

18 

0 

HRS  USE 

2.1 

12.9 

12.0 

.0 

Figure  4.  Example  Audit  Trail  Data  Summary 
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should  eventually  be  at  an  acceptably  low  rate.  It  should  hi  ,n>i.d  ih.it 

the  data  collected  did  support  the  audit  trail  and  regression  ma ' .  • 

proof  that  the  data  quality  can  be  made  to  be  acceptable.  Consult.- 1  .it  i mi 
of  the  adequacy  of  the  data  with  regard  to  information  content  is  f;i.en  in 
the  following  section. 

INFORMATION  GENERATION 

The  other  aspect  of  performance  measurement  to  be  considered  here  is 
that  of  generating  the  information  necessary  for  the  conduct  of  training 
and  associated  activities.  The  information  requirements  will  be  discussed 
first  to  identify  the  role  measurement  is  to  play  for  the  ISS,  and  following 
that,  specific  performance  measurement  design  issues  will  be  discussed. 

It  should  be  noted  that  the  ISS  design  occurred  circa  1978  while  the 
discussion  presented  in  this  report  draws  upon  new  data,  new  concepts  and 
experience  which  were  not  available  then.  It  should  be  remembered  that  the 
ISS  was  designed  within  tight  real-life  contraints,  and  that  the  finished 
product  did,  in  general,  function  satisfactorily.  Furthermore,  while  some 
improvements  may  appear  to  be  quite  desirable,  detailed  analysis  will  he 
required  to  determine  engineering  and  economic  feasibility. 

INFORMATION  REQUIREMENTS.  The  ultimate  purpose  of  performance  measurement 
is  to  provide  information  in  support  of  some  activity  or  decision;  It  is 
certainly  not  an  end  unto  Itself.  It  is  desirable,  therefore,  to  c onsldei 
the  end  uses  before  attempting  to  assess  the  specific  details  of  perfor¬ 
mance  measurement  for  instructional  support. 

Users  of  Performance  Measurement.  The  end  users  of  an  instructional  sup¬ 
port  system  include,  of  course,  the  instructors  and  students,  but  also 
developers  of  new  or  revised  training  material,  system  test  and  maintenance 
personnel,  training  system  managers,  system  developers,  and  the  system 
itself  for  various  automatic  functions.  Each  of  these  has  individual 
requirements  for  information  from  the  training  system,  and  the  performance 
measurement  subsystem  must  satisfy  these  needs.  An  extensive  analysis  oi 
the  needs  of  each  is  not  appropriate  here,  but  it  should  be  cleat  that  a 
system  that  only  attends  to  the  needs  of  the  student  and  instructor  will 
fall  short  of  the  possible  ISS  potential. 

Modes  of  Operation.  The  principal  modes  of  operation  for  the  ISS  ate 
CANNED,  ISEL  and  SST;  however,  the  ISS  must  permit  the  creation  and  modifi¬ 
cation  of  task  modules,  the  creation  and  test  of  CANNED  and  ISEL  programs, 
the  training  of  new  instructors  in  the  use  of  the  ISS  and  applications  for 
training,  and  the  test  of  the  system  to  ensure  that  it  is  functioning  pro¬ 
perly.  While  the  latter  are  not  the  primary  modes  of  the  ISS,  performance 
measurement  must  come  into  play  for  each,  and,  in  the  case  of  system  tests, 
the  performance  measurement  itself  should  be  included  in  testing  for  proper 
operation. 

Functions.  An  instructor  will  probably  begin  an  ISS  session  reviewing  stu¬ 
dent  progress  by  perusing  reports  generated  by  the  performance  measurement 
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system.  Then  the  ISS  must  support  briefing  and  debriefing,  and  at  times 
provide  these  functions  automatically  for  the  student  to  receive 
appropriate  feedback  when  an  instructor  is  not  present.  The  ISS  must  pro¬ 
vide  simulator  setup  and  initialization,  which  through  the  task  module 
definitions  define  the  performance  measurement  to  be  applied  during  each 
segment  of  training.  In  addition  to  performance  measurement,  per  se,  the 
ISS  must  provide  for  the  automatic  generation  of  grades  and  the  automatic 
generation  of  a  prescription  for  future  training. 

Training  System.  The  ISS  is  part  of  a  larger  training  system  and  should 
provide  information  for  training  management,  Instructional  design  and 
research.  The  performance  measurement  system  will  therefore  need  to  pro¬ 
vide  backup  Information  to  grades,  detailed  archival  data  for  subsequent 
analysis,  adaptive  logic,  and  other  information  for  the  management  and  con¬ 
tinued  development  of  the  ISS. 

A  Context  For  Assessment.  The  above  presentation  of  users,  modes,  functions 
and  training  system  environment  is  intended  to  provide  a  context  for  perfor¬ 
mance  measurement  assessment  within  the  ISS.  However,  it  should  be  noted 
that  the  specific  implementation  of  the  ISS,  which  Is  the  focus  of  this 
report,  may  not  have  addressed  the  entire  context  due  to  time  and  budget 
constraints.  This  reality  will  be  put  aside  and  for  present  purposes,  the 
full  potential  of  the  ISS  will  be  taken  as  a  context  for  assessment. 

PERFORMANCE  MEASUREMENT  DESIGN  ISSUES.  Major  measurement  system  functions, 
as  depicted  in  Figure  5,  are:  (a)  sampling,  (b)  segmentation,  (c)  scoring, 
(d)  performance  alerts,  (e)  grading  and  (f)  condensed  time  history  record¬ 
ing.  These  functions  must  answer  to  the  requirements  for  information  as 
outlined  previously.  The  structure  presented  in  Figure  5  is  intended  as 
an  aid  for  comparison  against  generic  measurement  requirements.  (Note 
that  the  ISS  measurement  is  accomplished  within  concurrent  task  modules; 
the  actual  Implementation,  while  Including  the  functions  shown  in  Figure 
5,  is  not  Implemented  as  the  blocks  shown  in  Figure  5.) 

Parameter  Sampling.  The  measurement  process  depends  on  sufficiently  fine¬ 
grained  sampling  of  system  state  variables  so  that  the  essential  character¬ 
istics  can  be  reproduced  and  reflected  in  computed  measures.  The  ISS  design 
provides  for  three  types  of  sampling  categories:  (a)  continuous,  wherein 
a  mission  segment  has  predefined  start  and  stop  conditions  and  the  associ¬ 
ated  parameters  are  sampled  1  to  20  times  per  second,  (')  monitor,  wherein 
a  parameter  is  sampled  during  the  time  that  it  is  outside  a  predefined 
limit,  and  (c)  snap  shot,  wherein  a  single  parameter  is  sampled  once. 

This  structure  provides  an  excellent  basis  for  subsequent  performance 
measurement  calculations  and  should  be  sufficient  for  future  requirements 
across  the  full  spectrum  of  users,  modes,  functions  and  management  needs. 
Past  experience  Indicates  that  the  sampling  rate  provided  is  more  than  suf¬ 
ficient.  Most  flight  parameters  need  only  be  sampled  two  times  per  second, 
while  only  values  such  as  pilot  stick  Inputs,  formation  deviations  and  air- 
to-air  mission  parsmeters  need  to  be  measured  up  to  10  times  per  second, 
and  final  segments  of  air-ground  weapons  delivery  require  sampling  at  20 
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times  per  second.  In  fact,  for  mass  data  storage  for  research  purposes, 
consideration  should  be  given  to  compressing  or  packing  some  of  the  Infor¬ 
mation  (such  as  recording  only  snapshots  when  a  discrete  change  occurs). 

Segmentation.  The  start  and  stop  of  segments  of  maneuvers  for  measurement, 
and  the  scoring  of  performance  within  these  segments,  are  included  as  part 
of  the  task  module  definitions.  Start  and  stop  of  measurement  segments  are 
defined  in  terms  of  discrete  events  such  as  "weight  off  wheels"  and  "weight 
on  wheels”.  While  this  method  is  often  adequate,  it  falls  short  in  terms 
of  permitting  unequivocal  definition  for  start  and  stop  of  basic  segments 
such  as  simple  turns.  Logic  based  on  two  parameters  is  generally  desirable. 
Further,  a  better  algorithm  results  when  a  window  (5-15  seconds)  of  para¬ 
meter  storage  is  maintained,  so  that  the  logic  permits  examining  trends. 
This  will  permit  the  initiation  of  a  turn  to  be  distinguished  from  a 
course  correction  maneuver.  Further  simplification  can  result  when  the 
next  segment  is  always  known;  then  it  may  be  possible  to  eliminate  start 
logic  by  using  the  method  that  the  next  segment  is  known  and  that  it  can 
start  when  the  previous  segment  end  conditions  are  met.  Apparently  the 
ISS  start/stop  logic  did  work  for  the  implemented  task  modules  during  the 
tests  performed  for  this  study.  On  the  other  hand,  experience  has  indi¬ 
cated  that  multi-parameter  logic  and  the  windowing  technique  are  sometimes 
required.  Therefore,  it  is  recommended  that  consideration  be  given  to 
start/stop  logic  refinement. 

The  ISS,  as  implemented,  permits  the  definition  of  measurement  as  part 
of  each  task  module;  however,  measurement  may  require  multiple  segments 
within  a  single  task  module.  For  example,  each  module  for  aontinuous  flight 
control  should  be  divided  into  at  least  three  parts:  (a)  capture  (e.g., 
transition  or  intercept)  of  a  steady  state  course,  (b)  course  tracking 
(e.g.  regulator  behavior),  and  (c)  departure  from  course,  or  recovery 
(another  transition).  Such  subdivision  of  global  task  modules  will  be 
necessary  to  produce  performance  measurement  that  reflects  fine-grain  pilot 
performance. 

Scoring.  ISS  scoring  for  continuous  performance  frequently  invokes  the 
concept  of  measurement  of  deviation  from  a  fixed  value,  or  exceeding  a 
fixed  value.  Figure  6  depicts  a  phase-plane  approach  derived  from  control 
system  theory,  which  treats  limits  in  terms  of  two  parameters  (defining  a 
plane)  rather  than  attempting  to  treat  performance  in  terms  of  one  para¬ 
meter  at  a  time.  Use  of  the  phase  plane  involves  a  position  parameter  and 
the  corresponding  rate  parameter  (e.g.,  altitude  and  rate-of-climb) .  In 
these  terms,  the  plane  can  be  used  to  define  a  central  area  of  acceptable 
performance,  with  the  remainder  divided  into  zones  in  which  the  state  is 
either  converging  to  the  proper  level  or  diverging.  Only  the  diverging 
zone  represents  what  is  ordinarily  considered  deviant  performance,  and 
thus  should  be  the  basis  for  scoring.  The  phase-plane  approach  is  recom¬ 
mended  for  consideration  in  future  ISS  development. 

Since  students  and  instructors  are  primary  users,  it  is  important 
that  the  measurement  be  displayed  to  them  in  terms  that  are  easily  inter¬ 
preted  by  them.  The  Implemented  ISS,  however,  presents  such  scores  as  the 
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Figure  6.  Example  Phase  Plane  Representation. 
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Root-Mean-Squared  (RMS)  Error  which  are  not  part  of  the  normal  flight  ter¬ 
minology.  It  is  recommended  that  such  measures,  if  used,  be  used  as  the 
basis  for  generating  messages  which  are  presented  in  ordinary  pilot 
language  as  represented  by  the  usage  in  current  flight  training  manuals. 

It  is  believed  that  the  presentation  of  scores  in  terms  of  tolerance 
bands  about  desired  performance  is  a  scoring  method  that  is  easily  under¬ 
stood  by  pilots.  The  main  disadvantage,  however,  is  that  these  measures 
give  no  information  about  performance  above  and  below  the  tolerance  level. 
An  ill-chosen  tolerance  band  will  not  discriminate  between  different  stu¬ 
dents,  and  frequently  the  actual  performance  accuracy  is  lost.  An  alter¬ 
native  that  somewhat  offsets  this  criticism  is  to  use  multiple  tolerance 
bands,  and,  it  is  suggested  here  that  three  levels  of  performance  tolerances 
be  used  for  assessment  purposes.  This  approach  has  implications  to  scoring 
which  will  be  discussed  below.  The  first  tolerance  band  will  represent 
superior  performance,  the  second  will  represent  good  performance,  and  the 
third  will  represent  the  outer  limits  of  commonly  accepted  operational 
criteria.  Furthermore,  momentary  excursions  shall  be  permitted  by  other 
criteria  which  specify  the  amount  of  time  a  system  state  variable  can  be 
in  a  category  before  the  measurement  system  will  recognize  the  fact.  For 
measurement  development,  however,  the  RMS  measure  is  an  effective  global 
measure  of  performance. 

Procedures  Monitoring.  Measurement  of  pilot  performance  during  normal  and 
emergency  procedures  requires  sensing  of  a  large  number  of  pilot  devices. 
In  addition  to  the  necessary  switches  for  the  given  procedure,  the  nearby 
and  functionally  related  switches  must  also  be  sensed  in  order  to  measure 
erroneous  inputs.  There  are,  of  course,  some  pilot  actions  which  cannot 
be  readily  sensed  (e.g.  communications)  and  Instructor  monitoring  of  these 
is  necessary  to  augment  the  automated  monitoring.  Each  activation  must  be 
checked  against  the  allowable  sequence.  For  example,  it  may  not  matter 
whether  a  given  procedural  event  occurs  before  or  after  some  other  events, 
but  must  occur  before  or  after  some  specific  events.  Furthermore,  there 
may  be  certain  contingencies  for  some  events;  for  example,  application  of 
nose-wheel  steering  on  takeoff  may  be  optional  prior  to  15  knots,  mandatory 
from  15  to  85  knots,  and  not  to  be  used  above  100  knots. 

In  addition  to  switch  position  and  sequencing,  the  performance  mea¬ 
surement  included  recognition  reaction  time  and  total  time  to  complete 
a  procedure.  Switch  positioning  could  be  weighted  as  mandatory,  critical 
or  optional.  Performance  measurement  therefore  included  percent  of  man¬ 
datory  actions  taken  and  percent  of  optional  actions.  A  check  list  or 
malfunction  procedure  could  be  presented  to  the  instructor  and  each  step 
in  tie  procedure  noted  as  It  Is  accomplished.  The  time  for  each  step  is 
displayed  and  errors  of  omission  or  comission  result  in  the  display  of 
diagnostic  messages. 

Proctiiurei.  ronitoring  of  the  sophistication  and  complexity  developed 
in  the  Implemented  ISS  is  a  pioneering  effort  with  little  precedent.  For 
example,  measurement  of  errors  of  comission  Is  an  Inherent  problem  since  all 
possible  switch  actions  often  c.'  not  be  scanned  within  available  computer 


51 


NAVTRAEQUIPCEN  81-C-0081-1 


time;  the  alternative  is  the  investment  of  large  amounts  of  analysis  to 
identify  reasonable  numbers  of  nearby  and  functionally  related  switches. 
However,  procedures  monitoring  is  one  of  the  strongest  features  of  the  ISS 
performance  measurement.  It  is  unfortunate  that  detailed  empirical  testing 
of  this  specific  feature  was  not  accomplished,  but  the  procedures  moni¬ 
toring  feature  appears  to  be  fundamentally  sound.  Procedures  monitoring 
is  a  performance  measurement  feature  which  can  contribute  significantly  to 
instructor  unburdening  and  is  worthy  of  additional  research  and  develop¬ 
ment  to  refine  and  verify  the  design. 

Performance  Alerts  And  Messages.  The  diagnostic  messages  presented  to  the 
instructor  during  student  training  are  practical  and  in  operational  lan¬ 
guage-  These  are  good  examples  of  the  position  advocated  above.  Perfor¬ 
mance  messages  should  provide  diagnostic  information  as  these  seem  to  do, 
and  should  enable  the  generation  of  remedial  strategies.  The  checklist 
displays  to  the  pilot  permit  the  instructor  to  follow  the  sequencing  of 
actions  and  note  errors  of  omission  and  commission.  These  appear  to  be 
quite  satisfactory.  The  Inclusion  of  response  times  and  time  of  action  may 
or  may  not  be  useful  to  the  instructor,  but  may  be  of  value  for  research 
and  development  purposes. 

Grading.  Grading,  in  the  implemented  ISS,  is  in  terms  of  the  range  in 
which  a  computed  measure  falls.  As  the  grading  structure  is  displayed  to 
the  instructor,  and  should  be  understandable,  this  scheme  has  some  merit. 
However,  the  grade  is  based  on  gross  measures  which  may  not  be  very  sensi¬ 
tive  to  brief,  but  serious,  performance  deviations.  It  is  recommended  that 
a  grading  scheme  be  considered  based  on  the  three-tolerance  band  scheme 
described  above.  A  five-point  grading  scale  could  be  used  as  shown  in 
Table  9,  and  the  maneuver  grade  would  be  as  shown  in  Table  10.  Grading  and 
certain  performance  monitoring  functions  are  simplified  if  serious  errors 
are  always  monitored  (as  the  implemented  ISS  Is  capable  of  doing).  Fatal 
errors  should  be  displayed  to  the  instructor  as  exemplified  in  Table  11; 
these  errors  should  cause  an  automatic  failure  of  a  segment  and  the  maneu¬ 
ver. 

Condensed  Time  History  Recording.  The  implemented  version  of  the  ISS  does 
not  permit  additional  detailed  performance  analysis  during  debriefing,  nor 
does  it  permit  the  computation  of  detailed  additional  measures  at  a  later 
date  for  research  purposes.  The  capability  should  exist  so  that  detailed 
supplemental  performance  measures  can  be  calculated  during  debriefing  when 
the  instructor  is  attempting  to  provide  diagnosis  and  feedback  to  the  stu¬ 
dent.  Time  history  recordings  are  necessary  to  permit  these  analyses.  To 
permit  detailed  analysis  during  debriefing  the  time  history  data  must  be 
retained  in  the  ISS;  however,  it  can  be  subsequently  spooled  out  to  tape, 
or  some  other  convenient  form,  for  long-term  storage.  The  stored  data  must 
include  both  continuous  and  discrete  state  variables,  but  only  within  desig¬ 
nated  measurement  segments,  and  possibly  in  packed  format  (e.g.,  record  and 
time-stamp  data  only  when  discrete  changes  occur). 
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TABLE  9.  EXAMPLE  SEGMENT  GRADE  CRITERIA 


Performance 

Level 

Conditions 

Segment 

Grade 

EXCEPTIONAL 

Unusually  accurate  performance 

No  Measures 

Exceed  Tol  1 

5 

ABOVE  AVERAGE 

All  measures  well  within 
acceptable  tolerance 

All  Measures 
within  Tol  1  or  2 

No  Measures 

Exceed  Tol  2 

4 

AVERAGE 

All  measures  within 
acceptable  tolerance 

All  Measures  within 

Tol  1  or  2  and  3 

No  Measures 

Exceed  Tol  3 

3 

BELOW  AVERAGE 

All  measures  within 
acceptable  tolerance,  but 
very  close  to  the  limits 

No  Measures  within 

Tol  1  and  2 

No  Measures  exceed 

Tol  3 

2 

UNSATISFACTORY 

At  least  one  measure 

out  of  acceptable  tolerance 

Any  Measure  Exceeds 

Tol  3 

1 

SEGMENT  NOT  COMPLETED  OR 
FATAL  ERROR 
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TABLE  10.  EXAMPLE  MANEUVER  GRADE  CRITERIA 


Performance 

Segment 

Maneuver 

Level 

Grade 

Grade 

EXCEPTIONAL 

Unusually  accurate  Performance 

AVG*  >4.7 

5 

ABOVE  AVERAGE 

All  segments  well  within 
acceptable  tolerance 

4.7  >  AVG  >  3. 7 

4 

AVERAGE 

All  segments  within 
acceptable  tolerance 

3.7  >  AVG  >2.7 

3 

BELOW  AVERAGE 

All  segments  within 
acceptable  tolerance, 
but  very  close  to  limits 

2.7  >  AVG  >  2.0 

2 

UNSATISFACTORY 

Any  Segment  *  1 

1 

MANEUVER  NOT  COMPLETED 

OR  FATAL  ERROR 

Any  Segment  ■  0 

0 

*AVG  »  Sum  of  Segment  Scores  divided  by  Number  of  Segments  Flown. 
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TABLE  11.  EXAMPLE  FATAL  ERRORS 


Fatal  Error 

Excessive  high  or  low  Airspeed 

Excessive  Bank  Angle 

Excessive  high  or  low  Altitude 

Excessive  Rate  of  Descent 
(One  minute  to  live  rule) 

Excessive  Time  In  a  Segment 

Violation  of  Airspace  Limits 

Turning  In  wrong  direction 

Climb  or  Descent  In  wrong  direction 

Descent  below  MDA  or  DH 

Improper  radio  frequency  when 
critical 

TACAN  T/R  switch  not  set  when 
DME  Is  critical 

Improper  Altimeter  Secting 


Comment 

+  30  knots  from  desired 

+  30  degrees  from  desired 

+  250  feet  from  desired 

W  greater  than  altitude 
above  the  ground 

tbd,  each  segment 

tbd,  each  segment 

tbd ,  each  segment 

tbd,  each  segment 

-  50  feet  from  MDA  or  DH 

tbd,  each  segment 

tbd,  each  segment 

+  0.02  from  desired  to 
include  changeover  at 
18,000  feet 


Landing  Gear  up 


r 


NAVTRAEQUIPCEN  81-C -0081-1 


SUMMARY 

The  implemented  ISS  was  developed  under  stringent  time  and  budget, 
constraints,  which  Halted  ISS  performance  measurement  development. 
However,  training  and  research  applications  will  benefit  if  additional 
resources  can  be  provided  for  performance  measurement  development  to  be 
carried  beyond  the  present  stage.  One  must  consider  the  full  matrix  of 
(a)  users,  aodes,  functions,  and  training  manage?.,<.nt  interface  by  (b) 
measurement  characteristics  of  sampling,  segmentation,  scoring,  messages, 
grades  and  time  history  recording.  It  is  considered  most  important  that 
performance  measurement  development  be  advanced  In  the  following  areas: 
(a)  full  support  for  all  users,  (b)  segmentation  schemes  appropriate  to 
fine-grained  performance  measures,  (c)  scoring  to  reflect  detailed  perfor¬ 
mance  In  pilot  language  using  a  multi-level  tolerance  band  scheme,  and  (d) 
time-history  recording  to  permit  detailed  analyses  for  debriefing  and 
research.  Nevertheless,  the  existing  task  module  and  performance  measure¬ 
ment  scheme  does  have  a  sound  basis,  and  the  ISS  as  currently  developed 
can  provide  the  performance  measurement  needed  to  support  global  instruc¬ 
tional  and  research  evaluations. 

[ 
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SECTION  VI 
CONCLUSIONS 


This  report  has  presented  a  review  of  the  ISS  concept  from  three  dif¬ 
ferent  points  of  view;  however,  the  foremost  concern  at  this  point  is  for 
the  principal  objectives  of  the  ISS:  to  increase  simulator  utilization  and 
thereby  reduce  training  costs,  and  to  Improve  simulator  training  quality. 
These  topics  will  be  discussed  in  the  following  along  with  recommendations 
for  advancement  of  the  ISS  design  concept. 

UTILIZATION 

TRAINING  WITHOUT  AN  INSTRUCTOR.  Clearly  simulator  utilization  can  be 
increased  if  the  simulator  can  be  used  without  an  instructor  being  present 
at  each  training  session.  Furthermore,  the  needed  enhancements  also  pro¬ 
vide  distinct  advantages  for  the  training  normally  conducted  without  an 
instructor  present  (e.g.,  the  first  three  CQ  training  events),  student- 
scheduled  extra-time  training  events  and  use  of  the  trainer  by  fleet  per¬ 
sonnel.  The  ISS  permits  accomplishment  of  training  with  little  or  no 
assistance  from  either  an  instructor  or  a  training  device  operator,  and 
therefore  enables  use  of  the  simulator  facilities  with  less  constraint  by 
the  available  amount  of  instructor  labor. 

Of  course,  the  Instructor  serves  a  number  of  instructional  functions 
in  addition  to  operation  of  the  simulator,  and  feedback  to  the  student 
about  his  performance  is  an  important  function  to  be  automated  to  improve 
instructorless  training  in  the  simulator.  While  the  existing  ISS  perfor¬ 
mance  measurement  provides  much  good  information,  especially  for  procedures 
training,  additional  ISS  performance  measurement  capability  (e.g.,  tough 
instructorless  trial  checkrides)  is  required. 

CONCURRENT  BRIEF /DEBRIEF.  Unfortunately,  in  its  present  stage  of  imple¬ 
mentation,  the  ISS  was  little  used  for  briefing  and  debriefing.  However, 
the  ISS  concept  does  include  briefing  and  debriefing  which  can  provide 
needed  preparation  for  simulator  training,  much  out-of-simulator  learning, 
and  reinforcement  of  the  training  received.  Furthermore,  the  ISS  concept 
can  provide  for  briefing  and  debriefing  concurrent  to  simulator  training, 
so  that  the  simulator  can  be  used  by  one  student  while  another  is  being 
briefed  or  debriefed.  With  this  approach,  simulator  utilization  can  be 
Increased  without  sacrificing  the  important  training  advantages  of  brief¬ 
ing  and  debriefing.  Otherwise,  unfortunately,  there  is  pressure  to  skim 
through  the  briefing  so  that  simulator  time  is  not  wasted,  or  to  hurry  the 
debriefing  because  the  next  student  is  scheduled  to  get  on  the  simulator. 

TRAINING  QUALITY 

INSTRUCTOR  WORKLOAD.  Reduction  of  instructor  workload  should  result  in 
Improved  training,  for  if  the  instructor  is  busy  manipulating  the  simulator 
it  is  probable  that  critical  periods  of  performance  will  not  be  monitored. 
Use  of  the  ISS  could  reduce  Instructor  workload  in  several  ways:  automated 
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briefings  allow  a  student  to  brief  himself  with  little  or  no  assistance 
from  the  instructor.  Automated  performance  measurement  and  procedures 
monitoring  capability  of  the  ISS  frees  the  instructor  from  keeping  records 
and  notes  of  student  performance.  The  automated  voice  generation  feature 
frees  the  instructor  from  making  many  support  communications.  Further, 
automated  malfunction  insertion  frees  the  instructor  from  switch  manipula¬ 
tion.  Consequently,  the  instructor  should  be  unburdened  by  the  ISS  and 
permitted  to  place  greater  emphasis  on  instructional  duties. 

STUDENT  PERFORMANCE  DATA.  ISS  features  provide  the  instructor  with  more 
detailed  Information  about  student  performance  than  is  normally  available. 
Procedural  responses  and  performance  scores  for  each  task  module  are 
displayed.  This  information  can  be  used  in  real  time  to  modify  training 
and  provide  timely  feedback  to  the  student,  or  it  can  be  used  during 
debriefing  for  instructional  impact  at  that  time  and  for  determination  of 
a  prescription  for  the  next  training  session.  The  availability  of  infor¬ 
mation  for  Immediate  feedback  should  result  in  a  clear-cut  improvement  in 
training.  Further,  the  automated  record  keeping  of  ISS  would  allow  an 
instructor  the  opportunity  to  review  a  student's  previous  performance 
prior  to  a  training  session.  This  would  orient  training  toward  an  individ¬ 
ual  student's  needs,  especially  in  an  environment  where  a  student  is  not 
assigned  to  a  single  instructor. 

STANDARDIZATION  OF  TRAINING.  Standardization  of  training  can  be  increased 
through  ISS  features  such  as  CANNED  mode  missions,  standard  training  mod¬ 
ules,  standard  briefings  and  automated  measures  of  performance.  Unfortu¬ 
nately,  instructor  pilots  may  not  elect  to  use  some  of  these  features  in 
the  current  implementation,  or  may  not  use  some  features  due  to  inadequate 
training  in  use  of  the  ISS.  Also,  standardization  may  not  always  be  con¬ 
sidered  a  virtue  by  the  instructors,  who  may  consider  standardized  train¬ 
ing  as  inflexible.  While  not  resolving  this  issue,  the  ISS  concept  does 
allow  centralized  control  of  training  as  well  as  means  for  flexibility 
through  automation  and  manual  options.  Consequently,  the  ISS  provides  a 
tool  for  control  by  the  training  manager  and  should  allow  any  degree  of 
standardization  to  be  determined  through  future  research. 

HUMAN  ENGINEERING.  The  ISS,  as  currently  configured,  provides  a  compact 
workstation  which  is  easily  reached  and  viewed,  providing  a  comfortable  and 
functional  interface  for  the  Instructor.  Uncluttered  integrated  displays 
provide  information  in  a  convenient  form.  The  touch  panel  control  sur¬ 
faces  provide  an  environment  which  permits  design  not  permitting  illegal 
Inputs  (again,  except  for  parallax  problems).  While  some  areas  for  im¬ 
provement  remain,  the  ISS  workstation  is  a  major  step  toward  providing  an 
environment  to  enhance  instructor  effectiveness. 

RECOMMENDATIONS 

While  the  current  review  resulted  in  favorable  findings  for  the  ISS 
concept,  there  are  a  number  of  issues  to  be  remedied  for  the  current  ISS 
Implementation  and  features  of  the  concept  to  be  emphasized.  Recommendations 
for  selected  topics  are  presented  in  the  following  paragraphs. 
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BRIEF/DEBRIEF  STATION.  Inability  to  use  the  remote  ISS  station  for  brief¬ 
ing  was  clearly  a  major  deficiency  in  the  current  ISS  implementation.  The 
use  of  a  separate  station  for  briefing  and  debriefing  has  large  potential 
training  benefit,  and  as  pointed  out  above,  also  has  large  potential  impact 
on  equipment  utilization  if  the  brief /debrief  station  can  be  used  simul¬ 
taneously  with  the  simulator.  The  brief /debrief  station  can  be  used  to 
review  progress,  modify  the  lesson,  setup  parameters  for  the  simulator, 
review  lesson  scores  and  grades,  and  generate  prescriptions  of  next  lessons. 
These  activities  could  all  be  accomplished  while  another  student  is  per¬ 
forming  in  the  flight  simulator.  Upgrade  of  the  design  and  the  concept 
for  a  concurrent  brief /debrief  station  is  therefore  recommended. 

PERFORMANCE  MEASUREMENT.  Existing  ISS  performance  measurement  has  some 
features  which  provide  insufficient  detail  and  which  are  not  in  terms 
ordinarily  understood  by  pilots.  Furthermore,  the  performance  measurement 
system  does  not  permit  the  storage  of  detailed  time  histories  of  perfor¬ 
mance  for  analysis  during  debriefing  or  at  a  later  time  for  research  pur¬ 
poses.  Some  revision  to  the  scoring  scheme  and  a  mechanism  for  data 
storage  is  therefore  recommended.  These  improvements  will  be  required  to 
make  the  ISS  a  suitable  testbed  for  research  purposes. 

USER-SOFTWARE  INTERFACE.  The  ISS  software  design  is  based  on  a  structure 
of  task  modules  containing  detailed  information  for  describing  segments 
with  specific  learning  objectives.  While  this  appears  to  be  an  excellent 
design  choice,  the  development  of  task  module  definitions  proved  to  be  an 
extremely  labor-intensive  activity  which  could  only  be  performed  by  the 
design  contractor.  Consequently,  total  task  module  definition  for  all 
VF-124  lessons  did  not  occur  even  within  the  course  of  this  study.  The 
difficulty  in  defining  and  modifying  task  modules  can  be  a  serious  defi¬ 
ciency,  especially  in  a  dynamic  training  environment.  It  is  therefore 
recommended  that  a  better  task  module  definition  Interface  be  developed, 
one  which  would  permit  Instructional  designers  to  program  the  ISS  without 
assistance  from  the  design  contractor.  This  might  take  the  form  of  a  user 
interface  language  or  an  interactive  utility  program. 

INSTRUCTOR  TRAINING.  ISS  effectiveness  is  heavily  dependent  upon  the  ability 
of  the  instructors  to  use  it  as  designed.  And,  except  for  the  instructor¬ 
less  mode,  the  ISS  is  a  tool  intended  to  enhance  the  instructor's  effec¬ 
tiveness.  It  is  therefore  exceedingly  Important  that  adequate  Instructor 
training  be  given,  as  well  as  that  adequate  help  functions  be  implemented 
in  the  ISS. 

FLEXIBILITY.  While  standardization  is  desirable,  it  is  also  important 
that  instructors  have  sufficient  flexibility  to  modify  the  chosen  syllabus 
with  variations  to  enhance  training  or  efficiency.  The  existing  IOS  allowed 
more  flexibility  in,  for  example,  selecting  simulated  malfunctions  and  in 
slewing  the  aircraft  position  than  did  the  ISS.  Less  flexibility  to  alter 
training  contributed  to  requirements  for  increased  planning  in  ISS  train¬ 
ing.  It  is  therefore  recommended  that  ISS  flexibility  for  Instructional 
control  be  increased  (within  specified  syllabus  constraints). 
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HUMAN  ENGINEERING.  While  good  human  engineering  was  exhibited  in  the  ISS 
in  general,  specific  deficiencies  as  noted  in  this  report  require  correc¬ 
tion. 

ADDITIONAL  EVALUATION.  The  present  evaluation  should  be  viewed  as  a  pilot 
study  of  an  evolving  system.  An  additional  evaluation  of  the  ISS  will  be 
necessary  to  provide  a  conclusive  assessment  of  its  contribution  to  in¬ 
creased  simulator  utilization  and  Improved  training  quality.  However,  an 
additional  evaluation  will  be  worthwhile  only  if  a  number  of  conditions 
under  which  it  is  conducted  are  different  from  the  current  study.  Appendix 
C  provides  detailed  recommendations  for  subsequent  evaluation. 
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APPENDIX  A 

PRE-USE  INTERVIEW  GUIDE 


What  experience  have  you  had  with  computer  systems? 

a.  Large  data  processing  systems 

b.  Computer  supported  training 

c.  Home  computer  systems 

d.  Small  business  systems  (word  processors) 

e.  Other 

What  do  you  think  of  the  use  of  computer  systems  in  training? 

Do  you  think  computers  can  improve  the  quality  of  training? 

Do  you  think  modern,  highly  computerized  instructor  consoles  will  result 
in  improving  your  skills  as  an  instructor? 

In  your  opinion  will  computer  assists  to  training  be  difficult  to  use? 

Do  you  expect  any  problems  in  setting  up  Instructional  problems  on  highly 
computerized  Instructor  consoles? 

Do  you  think  computers  can  simplify  your  job  as  an  instructor? 

Do  you  think  it  is  reasonable  for  computer  models  of  controllers  to  "talk 
down  the  pilot”  for  carrier  landing  simulations? 

How  do  you  feel  machine  generated  student  performance  scores  will  be 
accepted  by: 

a.  Instructors  in  general 

b.  students 

c.  you? 

Do  you  prefer  guages  or  digital  readouts  for  functions  such  as  altitude, 
angle  of  attack,  etc.? 

Which  do  you  prefer:  Computer  touch  panel  or  mechanical  switches? 

Do  you  expect  ISS  will  give  more  complete  information  for  ins true tor/ stu¬ 
dent  use  compared  to  present  system? 
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APPENDIX  B 

POST-USE  OBSERVATION  AND  INTERVIEW  GUIDE 


Describe  Training  Observed 
Log-On 

Time  required  (measure) 

Procedural  sequence/organization 
Ease/dif f iculty 

Brief ing/Debrief ing 

Summarize  training  missions  involved 

Identify  modes  used  in  training  (CANNED,  ISEL,  STT) 

Describe  briefing  uses  of  ISS 
Describe  briefing  strengths/veaknesses 
Describe  debriefing  uses  of  ISS 
Describe  debriefing  strengths/weaknesses 

CANNED  Mode  Training 

When  used  (describe  student  and  training  mission) 

Training  problem  set-up  (ease/difficulty;  describe  specific  events) 
Initial  set-up 
Making  revisions 
Shifting  to  other  modes 
When  and  why 

Ease/diff iculty  (describe  specific  events) 

ISEL  Mode  Training 

When  used  (describe  student  and  training  mission) 

Training  problem  set-up 
Initial  set-up 
Making  revisions 

Task  modules  used  (relative  frequency) 

Shifting  to  other  modes 
Why  and  when 
Ease/difficulty 

STT  Mode  Training 

When  used  (describe  student  and  training  mission) 

Training  problem  set-up 

Task  modules  used  (relative  frequency) 

^tutting  to  other  modes 
When  and  why 
Ease/difficulty 
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Performance  Monltoring/Measurement  (Uses  and  Values) 

Procedure  sequence  and  timing  display 

Summary  scores 

First  level  score  detailing 

Flight  profile  graphics  display 
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APPENDIX  C 

RECOMMENDATIONS  FOR  SUBSEQUENT  EVALUATION 


Several  recommendations  for  a  subsequent  evaluation  of  the  ISS  are 
provided  below: 

•  A  pre-evaluation  period  should  be  established  to  ensure  that  the 
ISS  is  ready  to  be  evaluated  and  to  ensure  that  instructors  are 
adequately  trained  before  the  evaluation  begins. 

•  The  data  collection  period  should  be  long  enough  to  collect  data 
for  approximately  15  students  for  each  control  and  experimental 
group  contained  in  the  study.  Current  estimates  suggest  that  about 
9  months  would  be  required. 

e  The  ISS  should  be  developed  to  train  10-15  Device  2F95  events.  To 
the  extent  possible  all  of  the  events  for  a  given  stage  of  training 
(e.g.  FAM,  CQ,  etc.)  should  be  developed  for  ISS  training.  Other¬ 
wise  simulator  training  for  each  stage  of  training  consists  of  a 
mixture  of  training  conducted  with  the  IOS  and  the  ISS.  This  makes 
it  difficult  to  assess  the  contribution  of  ISS  training  to  student 
performance. 

•  ISS  training  modules  should  be  fully  developed  and  debugged  before 
the  evaluation  begins.  Moreover,  no  major  changes  should  be  made 
to  these  modules  during  the  evaluation. 

•  Instructor  training  should  be  completed  before  the  evaluation  be¬ 
gins-  It  may  be  advisable  to  select  a  small  group  of  instructors 
and  train  them  to  operate  the  ISS,  rather  than  trying  to  train  all 
of  the  instructors  in  the  squadron. 

•  Instructor  training  should  include  some  instruction  on  how  to  take 
advantage  of  the  instructional  capability  of  the  ISS. 

•  A  functional,  remote  (off-line)  ISS  instructor's  console  should  be 
available. 

•  Sufficient  time  should  be  scheduled  for  instructors  to  complete 
simulator  training,  including  a  designated  briefing  and  debriefing 
period.  This  would  provide  instructors  with  a  scheduled  opportun¬ 
ity  to  use  the  brief lng/debrlef ing  capability  of  the  ISS. 

a  Measures  of  student  performance  need  to  be  expanded  or  complemented 
with  additional  measures.  The  development  of  specific  behavioral 
objectives  which  the  ISS  is  expected  to  achieve  provides  one  pos¬ 
sible  solution;  the  development  of  a  more  sensitive  instructor  grad¬ 
ing  scale  provides  another. 

Although  these  recommendations  are  specific  to  an  evaluation  of  the 
ISS,  they  do  provide  guidance  for  training  effectiveness  evaluations  of 
other  instructional  support  devices. 
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Federal  Aviation  Administration 
FAA  NAFEC  ANA-230  Bldg  3 
Atlantic  City,  NJ  08405 

Dr.  Bruno  Beek 
Rome  Air  Development  Center 
Griffiss  Air  Force  Base 
Rome,  NY  13441 

LT  Jeff  Woodard 
RADC/IRAA 
Griffiss  AFB 
Rome,  NY  13441 


Comnanding  Officer 

Navy  Fleet  Material  Support  Office 

P.0.  Box  2010 

Attn:  Ralph  Cleveland,  Code  9333 
Mechanicsburg,  PA  17055 

Mr.  Leahmond  Tyre 

Fleet  Material  Support  Office 

Code  9333 

Mechanicsburg,  PA  17055 

CDR  P.  M.  Curran 
Office  of  Naval  Research 
(Code  270) 

800  North  Quincy  Street 
Arlington,  VA  22217 

Mr .  Robert  Larr 

Naval  Air  Development  Center 

Code  8143 

Warminster,  PA  18974 

Dr.  Christian  Skriver 
Naval  Air  Development  Center 
Code  6021 

Warminster,  PA  18974 

Mr.  William  E.  Gibbons 
Naval  Air  Development  Center 
Warminster,  PA  18974 

Mr.  Frank  R.  Previti 
Naval  Air  Development  Center 
Mail  Code  4043 
Warminster,  PA  18974 

Dr.  Julie  A.  Hopson 

Naval  Air  Development  Center 

Code  6021 

Warminster,  PA  18974 
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LCDR  Steve  Harris 

Naval  Air  Development  Center 

Code  6021 

Warminster,  PA  18974 

Charles  R.  meek,  Jr. 

United  States  Postal  Service 
Process  Control  Systems  Test  Facility 
9201  Edgeworth  Drive 
Washington,  DC  20027 

Mr.  Ben  Wallis 
Computer  Analyst 

David  Taylor  Naval  Ship  R&D  Center 
Bethesda,  MD  20084 

OUSDR&E  (R&AT)  (E&LS) 

CDR  Paul  R.  Chatelier 
Washington,  DC  20301 

Chief  of  Naval  Operations 
OP-39T 

Washington,  DC  20350 

Mr.  Hal  Murray 

Naval  Air  Systems  Command 

Code  53343B 

Building  JP-2,  Room  610 
Washington,  DC  20360 

Naval  Air  Systems  Command 
Code  5313A 

Attn:  LT  Thomas  M.  Mitchell 
Washington,  DC  20361 

Commander 

Naval  Air  Systems  Command 
Air  340F 

Washington,  DC  20361 
Commander 

Naval  Air  Systems  Command 
AIR  413F 

Washington,  DC  20361 


Mr.  Ernest  E.  Poor 
Naval  Air  Systems  Command 
Air  413B 
Room  336 

Washington,  DC  20361 
F.  Leuking 

Naval  Air  Systems  Command 
AXR-360A 
JP-1,  Room  612 
Washington,  DC  20361 

CDR  Richard  S.  Gibson 
Bureau  of  Medicine  and  Surgery 
Head,  Aerospace  Psychology  Branch 
Code  3C13 

Washington,  DC  02372 

Dr.  Sam  Schiflett 
Naval  Air  Test  Center 
SY  721 

Patuxent  River,  MD  20670 

Director,  National  Security  Agency 
9800  Savage  Road 
Attn:  T.  W.  Page,  R54,  FANX  II 
Ft.  George  G.  Meade,  MD  20755 

Dr.  John  F.  Boehm 

Director,  National  Security  Agency 

9800  Savage  Road 

Attn:  R-542 ,  Boehm 

Ft.  George  G.  Meade,  MD  20755 

Mr.  Harold  C.  Glass 
US  Postal  Lab 
11711  Parklawn  Drive 
Rockville,  MD  20852 

Mr.  John  T.  Master son 
US  Postal  Lab 
11711  Parklawn  Drive 
Rockville,  MD  20852 


4  of  9 


NAVTRAEQUIPCEN  8I-C-0081-1 


Abraham  Tersoff  Gen.  Manager 
US  Postal  Service  Res.  Center 
11711  Parklawn  Drive 
Rockville,  MD  20852 

Dr.  Tice  De  Young 

US  Army  Engineer  Topographic  Labor¬ 
atories  Research  Institute 
Ft.  Belvoir ,  VA  22060 

Defense  Adv.  Research  Projects  Agency 
Information  Processing  Tech.  Office 
1400  Wilson  Boulevard 
Arlington,  VA  22209 

Defense  Adv.  Research  Projects  Agency 
Cybernetics  Technology  Office 
1400  Wilson  Boulevard 
Arlington,  VA  22209 

Defense  Adv.  Research  Projects  Agency 
Infor.  Processing  Techniques  Office 
1400  Wilson  Boulevard 
Arlington,  VA  22209 

Chief  of  Naval  Research 
Code  458 

800  N.  Quincy  St. 

Arlington,  VA  22217 

Office  of  Naval  Research  (Code  221} 
Dir. ,  Electromagnetic  Technology 
800  N.  Quincy  St. 

Arlington,  VA  22217 

Mr.  Jerry  Malecki 
Office  of  Naval  Research 
Code  455 

800  N.  Quincy  St. 

Arlington,  VA  22217 

Mr.  Gordon  D.  Goldstein 
Office  of  Naval  Research 
Code  437 

800  N.  Quincy  St. 

Arlington,  VA  22217 


Mr.  J.  Trimble 

Office  of  Naval  Research 

Code  240 

800  N.  Quincy  St. 

Arlington,  VA  22217 

Dr.  Henry  M.  Halff 
Office  of  Naval  Research 
Code  458 

Arlington,  VA  22217 

Dr.  Henry  J.  Dehaan 
US  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Charles  Hartz 
FLECOMTRACENLANT 
Code  02A 

Virginia  Beach,  VA  23416 

Mr.  Joe  Dickinson 
US  Army  Applied  Tech.  Lab 
Ft.  Eustis,  VA  23662 

CAPT  Leslie  K.  Scofield 
Directorate  of  Training 
US  Army  Signal  Center 
Ft.  Gordon,  GA  30905 

Dr.  James  D.  Mosko 
Naval  Aerospace  Medical  Res.  Lab. 
Acoustical  Sciences  Division 
Code  L348 

Pensacola,  FL  32508 

MAJ  Neal  Morgan  (LGY) 

Air  Force  Logistics  Mgmt  Center 
Bldg.  205 

Gunter  AFB,  AL  36114 

Dr.  Michael  G.  Sanders 
US  Army  Res.  Inst.  Field  Unit 
P.0.  Box  476 
Ft.  Rucker,  AL  36362 
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CW02  Ray  Priest 

Naval  Air  Tech.  Training  Center 

Code  7411 

NAS  Memphis  (85) 

Millington,  TN  38054 

Lt.  Col.  Robert  O'Donnell 
6570  ARML-HEA 
Wright-Patterson  AFB 
Dayton,  OH  45322 

Mr.  Eric  Werkowitz 
AFFDL/FGR 

Wright  Patterson  AFB,  OH  45433 
ASD/AXA 

Attn:  N.  R.  Vivians 
Wright-Patterson  AFB,  OH  45433 

CAPT  Barry  P.  McFarland 
US  Air  Force 
ASD/ENECH 

Wright-Patterson  AFB,  OH  45433 

Mr.  Don  F.  McKechnie 
Research  Psychologist 
AFAMRL/HEF 

Wright-Patterson  AFB,  OH  45433 

CAPT  Ronald  J.  Marine 
US  Air  Force 
ASD/AER-EX 

Wright-Patterson  AFB,  OH  45433 

Mr.  John  Courtright 
AFAMRL/HEG 

Wright-Patterson  AFB,  OH  45433 

Mr.  Don  Monk 
AMRL/HED 

Wright-Patterson  AFB,  OH  45433 


CAPT  Vince  Mortimer 
AFAMEL/BBM 

Wright-Patterson  AFB,  OH  45433 

Noel  P.  Schwartz 
AFHRL/ASM 

Advanced  Systems  Division 
Wright-Patterson  AFB,  OH  45433 

Mr.  Timothy  Theis 
ASD/RWR 

Building  20,  Aero.  B 
Wright-Patterson  AFB,  OH  45433 

Chief  of  Naval  Education  &  Training 
Liaison  Office 
Human  Resource  Laboratory 
Flying  Training  Division 
Williams  AFB,  AZ  85224 

Robert  F.  Lawson,  CDR,  USN  (Ret) 
Naval  Applications  Engineer 
ONR  Scientific  Department 
1030  E.  Green  Street 
PaSadena,  CA  91106 

Dr.  Keith  Bromley 

Naval  Ocean  Systems  Center 

Code  8111 

San  Diego,  CA  92152 

Mr.  Warren  Lewis 
Naval  Ocean  Systems  Center 
Human  Engineering  Branch 
Code  8231 

San  Diego,  CA  92152 

Mr.  Harry  A.  Whitted 
Code  8235 

Naval  Ocean  Systems  Center 
271  Catalina  Boulevard 
San  Diego,  CA  92152 


Mr.  Thomas  J.  Moore 
AFAMRL/BBA 

Wright-Patterson  AFB,  OH  45433 
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Dr.  Robert  A.  Wisher 
Navy  Personnel  Research  and 
Development  Center 
Code  P309 

San  Diego,  CA  92152 

Mr.  Melvyn  C.  Moy 
Navy  Personnel  Res.  &  Dev.  Center. 
Information  &  Decision  Processes 
Code  305 

San  Diego,  CA  92152 
John  Silva 

Naval  Ocean  Systems  Center 
Code  823 

San  Diego,  CA  92152 

Mr.  Gary  Poock 
Naval  PG  School 
Code  55PK 

Monterey,  CA  93940 

Mr.  Clayton  R.  Coler 
Research  Scientist 
NASA  Ames  Research  Center 
Mail  Stop  239-2 
Moffett  Field,  CA  94035 

Hallie  M.  Funkhouser 
Technical  Assistant 
NASA,  Ames  Research  Center 
Mail  Stop  293-3 
Moffett  Field,  CA  94035 

Kinga  M.  Perlacki 

NASA,  Ames  Research  Center 

Mail  Stop  239-2 

Moffett  Field,  CA  94035 

Dr.  Edward  Muff 

Chief,  Helicopter  Human  Factors  Ofc 
Mail  Stop  239-21 
NASA,  Ames  Research  Center 
Moffett  Field,  CA  94035 


Dr.  Robert  P.  Plummber 

Asst.  Prof.,  University  of  Utah 

NASA,  Ames  Research  Center 

Mail  Stop  239-2 

Moffett  Field,  CA  94035 

Mr.  Robert  H.  Wright 

Research  Psychologist 

Army  Research  Inst.  Field  Unit 

P.0.  Box  476 

Ft.  Rucker,  AL  56362 

Commander 

Naval  Air  Systems  Command 
Air  413G 

Washington ,  DC  20361 

Mr.  Ray  Satterfield 
Photographer 

Naval  Air  Development  Center 
Warminster,  PA  18974 

National  Aviation  Facilities 
Experimental  Center 
Library 

Atlantic  City,  NJ  08405 

Chief  of  Naval  Operations 
OP-593B 

Washington,  DC  20350 

Commander 

Naval  Air  Force 

US  Pacific  Fleet  (Code  316) 

NAS  North  Island 
San  Diego,  CA  92135 

Commander 

Training  Command 

Attn:  Educational  Advisor 

US  Pacific  Fleet 

San  Diego,  CA  92147 
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Commander 

Naval  Weapons  Center 
Code  3154 
Attn:  Mr.  Curtis 
China  Lake,  CA  93555 


Commander 

Naval  Air  Force 

US  Pacific  Fleet  (Code  342) 

NAS  North  Island 

San  Diego,  CA  92135 


Commanding  Officer 
Fleet  Anti-Submarine  Warfare 
Training  Center,  Pacific 
Attn:  Code  001 
San  Diego,  CA  92147 

commander 

Naval  Air  Test  Center 
CT  176 

Patuxent  River,  MD  20670 

Dr.  J.  D.  Fletcher 
Defense  Adv.  Research 
Projects  Agency  (CTO) 

1400  Wilson  Boulevard 
Arlington,  VA  22209 

Commanding  Officer 
Naval  Air  Technical  Training 
Code  104,  Building  S-54 
NAS  Memphis  (85) 

Millington,  TN  38054 

Mr.  Walt  Primas 

Chief  of  Naval  Operations 

OP-39T 

Washington,  DC  20350 

Chief  of  Naval  Operations 
OP-987H 

Attn:  Dr.  R.G.  Smith 
Washington,  DC  20350 


Commander 

Naval  Air  Development  Center 
Attn:  Code  6022 
Warminster,  PA  18974 

Commander 

Naval  Sea  Systems  Command 
Attn:  H.  Baker,  Code  6122 
Washington,  DC  20362 

Navy  Personnel  Research  and 
Development  Center 
Attn:  McDowell 
Library,  Code  P201L 
San  Diego,  CA  92152 

Chief  of  Naval  Operations 
OP-96 

Washington,  DC  20350 
Commandant 

US  Army  Field  Artillery  School 

ATSF-TD-T 

Mr.  Inman 

Ft.  Sill,  OK  73503 
Commandant 

US  Army  Field  Artillery  School 
Counterfire  Department 
Attn:  Eugene  C.  Rogers 
Ft.  Sill,  OK  73503 


Commander 

Naval  Air  Systems  Command 
Technical  Library 
AXK-950D 

Washington,  DC  20361 
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Director 

US  Army  Human  Eng.  Laboratory 
Attn:  DRXHE-HE  (KEESEE) 

Aberdeen  Proving  Ground,  MD  21005 

USAHEL/USAAVNC 

Attn:  DRXHF-FR  (Dr.  Hoffmann) 

P.O.  Box  476 

Ft.  Rucker,  AL  36362 

ASD/ENESS 

Attn:  R.B.  Kuhnen 

Wright  Patterson  AFB,  OH  45433 

Air  Force  Human  Resources  Lab 
AFHRL/LR 

Logistics  Research  Division 
Wright  Patterson  AFB,  OH  45433 

LT  Dave  Cooper 
AFHRL/OTT 

Wright  Patterson  AFB,  OH  45433 

US  Air  Force  Human  Resources  Lab 
AFHRL-IT  (Dr.  Rockway) 

Technical  Training  Division 
Loury  AFB,  CO  80230 

US  Air  Force  Human  Resources  Lid? 
TSZ 

Brooks  AFB,  TX  78235 

ASD/ENETC 

Mr.  R.G.  Cameron 

Wright  Patterson  AFB,  OH  45433 

Headquarters  34  Tactical  Airlift 
Training  Group/TTDI 
Little  Rock  AFB,  AL  72076 

Headquarters 

Air  Training  Conaand,  XPTX 
Attn:  Mr.  Goldman 
Randolph  AFB,  TX  78148 


Commanding  Officer 

Rome  Air  Development  Center 

Library  (TSLD) 

Griff iss  AFB,  NY  13446 

Director 

Air  University  Library 
Maxwell  AFB,  AL  36100 

Mr.  Harold  A.  Kottnumn 
ASD/YWE 

Wright  Patterson  AFB,  OH  45433 

Aeronautical  Systems  Div. 

USAF,  ASD/YWB'  (R.  Coward) 
Wright-Patterson  AFB 
Dayton,  OH  45433 

CDR  Charles  Theisen 
Lauren  Ridge 
R.D.  # 2,  Box  143-8A 
New  Hope,  PA  18938 

Chief 

ARI  Field  Unit 

P.  ,0.  Box  476 

Ft.  Rucker,  AL  36362 
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